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Executive Summary 

The eMINTS professional development programs are designed to help teachers learn how 
to integrate technology into their teaching, using instructional strategies that promote 
inquiry-based learning and encourage collaboration and community building among 
students and teachers. The eMINTS programs considered in this evaluation include: (1) 
eMINTS Comprehensive Professional Development (Comp PD), a two-year program 
consisting of approximately 250 hours of teacher professional development and support, 
for teachers in school-designated grades, including 10—12 Classroom Visits each year; (2) 
eMINTS4ALL, a two-year, 90-hour professional development program built for teachers in 
the grades above and below eMINTS Comp PD teachers, to support students for multiple 
years, including 8-9 Classroom Visits per year; and (3) Professional Development for 
Educational Technology Specialists (PD4ETS), a two-year program that prepares local 
district staff members to become eMINTS instructional specialists. 

Since 1999 the eMINTS National Center has engaged external evaluators to examine the 
program to understand its impact on schools, teachers, and students. In the 2005-2006 
school year, the eMINTS program added new professional development programs and 
expanded beyond the state of Missouri, where it began [as "enhancing Missouri’s 
Instructional Networked Teaching Strategies"). In order to help eMINTS ensure a high level 
of program quality as it scaled up, the Education Development Center, Inc.'s Center for 
Children and Technology (EDC/CCT) was brought in as the external evaluator in 2006 and 
charged with designing an evaluation that would create instruments and procedures to 
assess Program Fidelity and Teacher Mastery of eMINTS core concepts, and would examine 
the relationships among Program Fidelity, Teacher Mastery, and student achievement. 
Program Fidelity consists of two components: 

1. PD Fidelity, or how well the professional development addressed the key conceptual 
constructs of eMINTS 

2. Classroom Visits, or how much time instructional specialists spent on certain 
activities during their regular visits to participant classrooms. 


EDC/CCT’s evaluation strategy was designed to answer the following questions about the 
eMINTS Comprehensive and eMINTS4ALL professional development programs: 

Regarding Program Fidelity: 

• Are the current eMINTS professional development sessions faithfully addressing the core 
program constructs [PD Fidelity)? 
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• Are there differences in the level of PD Fidelity between professional development sessions 
delivered by eMINTS staff and those delivered by participants or graduates of the PD4ETS 
program? 

• What activities are specialists spending the most time on during their Classroom Visits? 

• Are there differences in the amount of time eMINTS staff and participants or graduates of 
the PD4ETS program spend on different activities during Classroom Visits? 

Regarding Teacher Mastery: 

• What are the baseline levels of Teacher Mastery of eMINTS concepts? 

• Are eMINTS participants mastering some concepts more successfully than others? 

Regarding Program Impact: 

• Does the level of eMINTS Program Fidelity have an impact on participating teachers’ 
mastery of the concepts presented in the professional development sessions? 

• Does the level of eMINTS Program Fidelity have an impact on the achievement of students 
in the classrooms of eMINTS teachers? 

• Does teachers' level of mastery of the program concepts have an impact on the achievement 
of student in their classrooms? 


EDC/CCT evaluators worked closely with eMINTS program staff to design Fidelity and 
Teacher Mastery instruments that were closely aligned with the core concepts about 
effective instruction and technology integration that serve as the foundation for all eMINTS 
programming. Data collected for the evaluation included: 


• Observational data from eMINTS professional development sessions 

• Records of Classroom Visits by eMINTS instructional specialists 

• Teacher artifacts (Lesson Plans, WebQuests, and Classroom Websites) submitted as part of 
eMINTS teachers' portfolios 

• Data from interviews conducted with 16 teachers one year after they completed the eMINTS 
program 

• Student assessment data from the Missouri Assessment Program (MAP) tests in 
Mathematics (MA), Communication Arts (CA), Science (SC), and Social Studies (SS) 


KEY FINDINGS FROM THIS EVALUATION OF EMINTS 


Regarding Program Fidelity, the findings suggest that there was a high level of fidelity to the 
core eMINTS concepts: 

• The majority of instructional specialists addressed many of the key concepts. 
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The majority of instructional specialists used the recommended instructional practices. 


• There were few differences in PD Fidelity between eMINTS staff and participants or 
graduates of the PD4ETS program. 

• During Classroom Visits, instructional specialists spent the most time working with 
teachers on Lesson Planning and Modeling Instruction and the least amount of time 
providing Technology Assistance and Problem Solving. 

• eMINTS staff spent more time than participants or graduates of the PD4ETS program on 
Lesson Planning during Classroom Visits. 

Regarding Teacher Mastery, the findings indicate a wide range in the levels of mastery of 
the eMINTS concepts, with certain concepts more successfully mastered than others: 


• Teachers displayed a high level of mastery of some core eMINTS concepts, such as 
integrating technology to support student learning and having students create authentic 
products to demonstrate their learning. 

• Teachers displayed a lower level of mastery of other core eMINTS concepts, such as 
designing instruction to address diversity and having students generate their own 
questions to guide their inquiry. 

Regarding Program Impact, our findings suggest that higher PD Fidelity is associated with 
greater Teacher Mastery of eMINTS concepts, and more time spent Lesson Planning in 
Classroom Visits is associated with greater Teacher Mastery as reflected in the Lesson 
Plans: 

• There was a significant, positive correlation between PD Fidelity and Teacher Mastery 
scores on the Lesson Plans teachers submitted in their portfolios. 

• There was a positive trend between PD Fidelity and Teacher Mastery on the WehQuests 
teachers submitted in their portfolios. 

• There was a significant, positive correlation between the amount of time teachers spent on 
Lesson Planning during Classroom Visits and the scores on the Lesson Plans they submitted 
as part of their portfolios. 

Also regarding Program Impact, our findings suggest that higher levels of Teacher Mastery 
of eMINTS concepts are associated with greater student achievement, higher levels of PD 
Fidelity are associated with greater student achievement, and more time spent on Lesson 
Planning during Classroom Visits is associated with greater student achievement: 


• There were significant, positive correlations between student MAP scores and Teacher 
Mastery on the Lesson Plan (in grades 3, 4, and 7), on the WebQuest (in grades 3 and 7), and 
on the Classroom Website (in grades 4, 5, and 7). 

• There were significant, positive correlations between PD Fidelity and student MAP scores in 
grades 3, 4, 5, and 8. 
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• There were significant, positive correlations between student MAP scores and Lesson 
Planning during Classroom Visits in grades 4, 5, and 8. 

Overall, this evaluation provides evidence that the eMlNTS program is being implemented 
with a high level of fidelity by both the eMlNTS staff and participants and the graduates of 
the PD4ETS program; that teachers are mastering some, but not all, of the core eMlNTS 
concepts; and that maintaining a high level of Program Fidelity is important for ensuring 
that teachers are mastering the core program concepts, which may then result in higher 
levels of student achievement. 
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Section I: Introduction 


eMINTS is a comprehensive program that offers a wide range of professional 
development options to teachers, administrators, technology specialists, and other 
educators. eMINTS professional development is designed to help teachers learn how 
to integrate technology into their teaching using instructional strategies that 
promote inquiry-based learning and encourage collaboration and community 
building among students and teachers. eMINTS began in 1999 as a professional 
development program for teachers in Missouri [it is an acronym for "enhancing 
Missouri’s Instructional Networked Teaching Strategies"), but it has since expanded 
to 11 more states and to Australia. The "flagship” professional development 
program is eMINTS Comprehensive PD [Comp PD), a two-year program consisting 
of approximately 250 hours of online and face-to-face teacher professional 
development and support, including 10-12 Classroom Visits each year by eMINTS 
instructional specialists. eMINTS4ALL was created as a professional development 
program for teachers in the grades above and below the grade taught by eMINTS 
Comp PD teachers, to help prepare students for and extend the kind of instruction 
they experience in eMINTS Comprehensive classrooms. eMINTS4ALL is also a two- 
year program, consisting of 90 hours of online and face-to-face instruction, with 8-9 
Classroom Visits each year by eMINTS instructional specialists. The PD4ETS 
program is a two-year program that prepares local district staff members to become 
eMINTS instructional specialists. Other programs include professional development 
for administrators, technology coordinators, and special education teachers, and 
continuing education for veteran eMINTS teachers. 

Evaluation has been an integral part of the eMINTS program since its inception. 
eMINTS program developers have commissioned external evaluators to take both 
formative and summative approaches to the evaluation of eMINTS. Data from these 
evaluations have been used to inform program improvement and to understand the 
impact of eMINTS on teacher participants and their students. 

From 1999 to 2005, Missouri’s Office of Social and Economic Data Analysis [OSEDA) 
served as the external program evaluator for eMINTS. This group conducted a wide 
range of evaluation activities, including focus groups and interviews with teachers, 
administrators, and parents; observations of eMINTS classrooms; surveys of 
teachers and students; and analyses of student assessment data from the Missouri 
Assessment Program [MAP) tests that compared outcomes for the students of 
eMINTS participants and comparison groups of students whose teachers had not 
participated in eMINTS [see http://www.emints.org/evaluation/index.shtml for a 
complete list of evaluation publications). These evaluation reports present a rich 
picture of the eMINTS program and the schools in which it has been implemented. 
Along with qualitative findings about such things as effective instructional practices 
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of eMINTS teachers, leadership styles of principals in schools participating in 
eMINTS, and professional development methodologies of educators in the eMINTS 
PD4ETS program, the OSEDA summative evaluations of student outcomes 
consistently found that eMINTS had a significant, positive impact on student 
achievement, especially for low-income. Special Education, and Title I students, in 
each of the five years for which they conducted these analyses. 

In 2006 the eMINTS program was scaling up dramatically, in both its programming 
(implementing more PD4ETS professional development sessions and adding the 
eMINTS4ALL program to its suite of professional development offerings) and its 
reach (expanding beyond Missouri to Utah, Maine, Nevada, Arkansas, Illinois, 
Oklahoma, Ohio, and Texas^). To ensure that they could maintain a high level of 
program quality as they scaled up the program, the eMINTS program staff 
recognized the need to revise their evaluation strategy, to build a better 
understanding of Program Fidelity and its relationship to the program's impact on 
teachers and students. They selected the Education Development Center, Inc.’s 
Center for Children and Technology (EDC/CCT) to develop instruments and 
procedures for collecting data about Program Fidelity and Teacher Mastery and to 
conduct an evaluation that looked at the relationships among Program Fidelity, 
Teacher Mastery, and student achievement.^ In this evaluation design. Program 
Fidelity consisted of two components: 

1. PD Fidelity, or how well the professional development addressed the key 
conceptual constructs of eMINTS 

2. Classroom Visits, or how much time instructional specialists spent on certain 
activities during their regular visits to participant classrooms. 

Teacher Mastery of concepts was determined through an analysis of some of the key 
artifacts (a Lesson Plan, a WebQuest, and a Classroom Website) teacher participants 
submitted in their program portfolios at the end of their two-year professional 
development experience. Student achievement was determined through an analysis 
of student MAP data. 

The new evaluation strategy was designed to answer the following questions about 
the eMINTS Comprehensive and eMINTS4ALL professional development programs: 


^ Since the 2005-2006 school year, eMINTS has further expanded to Alabama, Delaware, Minnesota, and 
New South Wales, Australia. 

^ Though EDC/CCT’s main evaluation responsibilities consisted of conducting this new approach to eMINTS 
evaluation, ECD/CCT evaluators also conducted an analysis comparing achievement on the 2005 MAP of 
students in eMINTS and non-eMINTS classrooms, and, consistent with the OSEDA evaluations, found the 
program had a significant, positive impact on student achievement. 
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Regarding Program Fidelity: 

• Are the current eMINTS professional development sessions faithfully addressing the 
core program constructs [PD Fidelity)? 

• Are there differences in the level of PD Fidelity between professional development 
sessions delivered hy eMINTS staff and those delivered by participants or graduates 
of the PD4ETS program? 

• What activities are instructional specialists spending the most time on during their 
Classroom Visits? 

• Are there differences in the amount of time eMINTS staff and participants or 
graduates of the PD4ETS program spend on different activities during Classroom 
Visits? 

Regarding Teacher Mastery: 

• What are the baseline levels of Teacher Mastery of eMINTS concepts? 

• Are eMINTS participants mastering some concepts more successfully than others? 

Regarding Program Impact: 

• Does the level of eMINTS Program Fidelity have an impact on participating teachers’ 
mastery of the concepts presented in the professional development sessions? 

• Does the level of eMINTS Program Fidelity have an impact on the achievement of 
students in the classrooms of eMINTS teachers? 

• Does Teacher Mastery of the program concepts have an impact on the achievement 
of student in their classrooms? 

This evaluation looked primarily at teachers who took part in the eMINTS 
Comprehensive and eMlNTS4ALL programs in Missouri between the fall of 2005 
and the spring of 2007. Data from the 2006-2008 cohort were collected as well and 
were included in our examination of PD Fidelity and the Classroom Visit records, for 
the purpose of instrument validation and refinement, and to have a larger dataset 
for determining the Program Fidelity baselines. The districts that participated in 
eMINTS during this time period did so in many different ways. Some districts used 
their own funds to pay for eMINTS professional development and made their own 
unique choices about which grades and teachers would participate. Some schools 
eligible for Title IlD federal funds applied to participate in eMINTS and were 
awarded competitive Title IID grants through the Missouri Department of 
Elementary and Secondary Education [DESE). DESE and the eMINTS National Center 
took a more systematic approach in 2005 than in previous years in awarding the 
grants to schools eligible for Title IID funds, encouraging whole grade levels and 
multiple grades in schools and districts to participate in the program. This approach 
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helped to establish larger and more robust support networks in schools and 
districts. However, it had implications for the kind of data analysis that could be 
conducted, which will be discussed below. Participants from the Title IID grant 
districts were required to submit teacher portfolios at the end of their eMINTS 
experience. Teachers from a few other districts also voluntarily submitted teacher 
portfolios and/or were included in the data collected from observations of eMINTS 
professional development sessions and classroom visit records. Because the 
Teacher Mastery data are derived from teacher portfolio artifacts, only data from 
teachers who submitted portfolios are included in the combined analyses that look 
at the relationships among Program Fidelity and Teacher Mastery, Program Fidelity 
and student achievement, and Teacher Mastery and student achievement. 

Fifty-five Missouri districts and one Arkansas district (shown in Table 1) 
participated in the evaluation project. The evaluation activities in which they took 
part are noted. 

Table 1: Participating Districts 


District 

Teachers and/or 
district eMINTS 
instructional 
specialist participated 
In observed session 

Teachers and/or 
district eMINTS 
Instructional 
specialist 
participated in 
classroom visits 

Teachers 

submitted 

portfolios 

Provided teacher 
rosters allowing 
student data to be 
analyzed 

Avenue City R-IX 

X 

X 



Bayless 

X 

X 



Bolivar R-l 

X 

X 



Cameron R-l 

X 

X 



Carrollton R-VII 

X 


X 

X 

Carthage R-IX 

X 

X 



Central R-lll 

X 

X 



Chillicothe R-ll 

X 

X 

X 

X 

Confluence Academy 

X 

X 



Corning School District (AR) 


X 



Crawford Co R-ll 

X 

X 



Dallas Co R-l 


X 



DeSoto 73 

X 

X 

X 


Fort Zumwalt R-ll 

X 


X 


Francis Howell R-lll 

X 

X 

X 

X 

Gasconade Co R-l 

X 

X 

X 

X 

Hartville R-ll 

X 

X 

X 

X 

Hazelwood 

X 

X 

X 


Hillsboro R-lll 

X 

X 

X 


Humansville R-IV 

X 

X 



Hurley R-l 

X 

X 



Jefferson City 

X 

X 

X 

X 

Joplin R-VIII 

X 

X 



Junction Hill C-12 

X 

X 
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Kingston K-12 

X 

X 

X 


Maplewood Richmond Heights 

X 

X 



Marion C. Early R-V 

X 

X 

X 


Monett R-l 

X 

X 

X 


Moniteau Co R-V 

X 

X 



New Franklin R-l 

X 

X 



Nixa R-ll 

X 

X 

X 


North Kansas City 74 

X 

X 

X 

X 

North Harrison R-lll 


X 



Norwood R-l 


X 



Oregon-Howell R-lll 

X 

X 



Perry County 32 

X 

X 

X 

X 

Poplar Bluff R-l 

X 


X 


Potosi R-lll 

X 

X 



Republic R-lll 

X 

X 

X 


Savannah R-lll 

X 

X 



Sedalia 200 

X 

X 



Seymour R-ll 

X 

X 

X 


Smithville R-ll 

X 

X 

X 

X 

Sparta R-lll 

X 

X 

X 

X 

Special School District 

X 

X 

X 


St. Charles R-VI 

X 


X 


St. James R-l 

X 

X 

X 


Tri-County R-VII 

X 

X 



Union R-XI 

X 

X 

X 


Valley Park 

X 

X 

X 


Warrensburg R-VI 

X 


X 


Webster Groves 

X 

X 

X 

X 

Wellston 

X 

X 

X 


West Plains R-VII 

X 

X 

X 


Wheatland R-ll 


X 

X 


Willow Springs R-IV 


X 




In Section II of this report, we describe the methods used to collect the evaluation 
data. In Section III, we describe the analyses that were conducted on the data and 
the creation of the comprehensive database. In Section IV, we present the findings 
from our analyses, and in Section V, we discuss the implications of this study. 
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Section II: Methods 


EDC/CCT evaluators used a variety of methods to obtain data to examine the 
implementation and impact of the eMlNTS program on teacher participants and 
their students. The evaluators collected data about eMINTS professional 
development sessions, visits to participants' classrooms, participants' mastery of the 
eMINTS concepts, and student achievement of participants' students. 

PD FIDELITY 

INSTRUMENT DEVELOPMENT 

Observational data from eMINTS professional development sessions were collected 
to determine the level of PD Fidelity experienced by eMINTS participants. Designing 
instruments to gather information about PD Fidelity required a close collaboration 
with the program developers, since they are the experts on what faithful program 
implementation entails. Our fidelity instrumentation development process 
comprised six steps: existing document review; initial core concept and individual 
item development; item review and validation; creation of instruments; training on 
instruments and assessment of inter-rater reliability; and data collection. 

EXISTING DOCUMENT REVIEW 

The first stage of development involved the collection and review of all existing 
eMINTS program documentation and identification of those elements that were 
deemed essential to the program. These materials included [but were not limited to] 
the program's lists of key competencies for eMINTS teachers, hallmarks of an 
effective eMINTS classroom, and the entire curricula for the PD program. 

INITIAL CORE CONCEPT AND INDIVIDUAL ITEM DEVELOPMENT 

Using the collected information described in the first step, the evaluation team 
identified and operationalized five core concepts of the eMINTS program: 

• Modeling Instruction 

• Community Building 

• Technology Utilization 

• Connection to Practice 

• Inquiry-Based Learning 

Each core concept was then further developed to include five to eight individual 
items. Specific criteria for the development of individual items were as follows: 

1. Items had to represent an observable or verifiable behavior, activity, or 
procedure. 
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2. Items had to be logically discrete from other items. 

3. Items had to be specific to the eMINTS program models. 

Together, they comprised our initial conceptual framework. 

ITEM REVIEW AND VALIDATION 

An adequately valid instrument will have: (1) a high level of face validity, meaning 
that it will be easily recognized and accepted by program developers as fitting with 
their understanding of the core features of the eMINTS program, and [2) a high level 
of content validity, meaning that experts with deep knowledge of the eMINTS 
curriculum and implementation model will accept the instrument’s 
characterizations of the curriculum and its key procedural and conceptual features. 
Thus, the third step in the development process involved the establishment of face 
and content validity. This was accomplished by engaging in a review process with 
the eMINTS leadership team, which served as our expert panel, whereby items were 
eliminated, modified, or added. Items were eliminated if they were not identified by 
the experts as essential to the eMINTS experience [across both the eMINTS 
Comprehensive and eMINTS4ALL implementation models], or they were modified 
to more accurately reflect the identified core elements of the program. New items 
that emerged during the review process that were deemed valuable and significant 
to the PD program by the experts were added to the framework at that point as well. 

CREATION OF INSTRUMENTS 

In the fourth step, the evaluation team created two instruments for gathering 
fidelity data from observations of eMINTS professional development sessions: a 
checklist to be completed at the end of the professional development session and 
another protocol that enables the observer to capture "snapshot” information about 
the professional development session at regular intervals [15 minutes] throughout 
the session. The utilization of this dual methodology provided information on 
fidelity of both structural elements of the eMINTS PD program [by reporting on the 
overall program delivery] and its dynamic processes [by reporting on the ways in 
which a program is being delivered]. Specifically, the Checklist instrument items 
focused more on the key conceptual and pedagogical elements of the program, 
whereas the Snapshot instrument focused primarily on the professional 
development process and facilitation strategies of the instructional specialist. 

After these two instruments were created, evaluation team members attended three 
professional development sessions [one they observed together, and two they 
observed independently]. The evaluators practiced using the instruments during 
these sessions, examined the data collected, and discussed how accurately the data 
reflected what they observed. These observations and post-observation analyses 
informed further refinement of the instruments and the data collection process. At 
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the conclusion of this process, the evaluators finalized the two instruments to reflect 
six core program elements, each including from 4 to 15 individual items (across 
both instruments): 

• Program Logistics and Planning 

• Modeling Instruction 

• Community Building 

• Technology Utilization 

• Connection to Practice 

• Inquiry-Based Learning 

As both instruments were derived from the same conceptual framework developed 
in earlier stages, they share the same core concepts and scoring mechanism 
(Yes/No). The two instruments also share their organizational format, as they are 
both divided into items that are evidenced by the facilitator, the participants, or the 
interaction of facilitator with participants. Individual items are not grouped by core 
element, to avoid response bias. 

TRAINING ON INSTRUMENTS AND ASSESSMENT OF INTER-RATER RELIABILITY 

Our next step involved the training of observers and assessment of inter-rater 
reliability on both protocols. The purpose of establishing inter-rater reliability was 
to assure the usability and consistency of the instruments' scoring system. During a 
day-long training session with a team of seven observers, scheduled a few weeks 
prior to the beginning of the fidelity observations, the evaluators reviewed the 
process of instrument development, introduced and carefully reviewed items on 
each of the instruments, and provided examples and clarifications for each item. 

This review process was accompanied by the viewing of 12 carefully selected video 
segments recorded during two separate eMINTS professional development sessions. 
Questions, clarifications, and opportunities for the observers to actually score 
segments were integral parts of this day-long training session. At the end of the 
training session, observers were shown two 30-minute video segments and were 
asked to score both of the instruments as they would do in a true observation. Inter- 
rater agreement calculated as the average percent agreement among all trained 
observers yielded an 83% and 88% agreement on the Checklist and Snapshot 
protocols, respectively. 

DATA COLLECTION 

Six eMINTS staff members conducted a total of 50 observations of 28 different 
instructional specialists facilitating four-hour eMINTS professional development 
sessions. Observers observed both eMINTS and eMINTS4ALL sessions for both Year 
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1 and Year 2 participants [see Table 2). Observers sat in on a range of modules 
(content] during the winter and spring of 2007. Some instructional specialists were 
observed more than once. Observation data were entered into a standard Excel form 
and submitted to the eMINTS Moodle [data sharing] site. These files were then 
downloaded by EDC/CCT evaluators and entered into a PD Fidelity database. 

Table 2: Number of professional development sessions observed, by 
program and teacher's year in that program 



eMINTS 

eMINTS4ALL 

Yearl 

11 

1 

Year 2 

28 

10 


CLASSROOM VISIT RECORD 

INSTRUMENT DEVELOPMENT 

An important element of the eMINTS program is the regular Classroom Visits that 
the eMINTS instructional specialists provide to the teachers during both years of the 
program. The eMINTS staff and evaluators wanted to understand what kind of 
classroom support eMINTS participants were requesting from their instructional 
specialists, and how often and for what length of time instructional specialists 
visited the teachers. For this reason, EDC/CCT evaluators developed a short Excel 
worksheet in which instructional specialists could record what they did during each 
Classroom Visit they made. The form asks for teacher demographic information 
[grade, school, district, subject taught, when applicable] and for instructional 
specialists to fill out the date of the visit; the amount of time they spent on specific 
activities [Modeling Instruction, Lesson Planning, Technology Assistance, Reflective 
Practice, Problem Solving, or other]; and the total amount of time spent on the visit. 
eMINTS instructional specialists reviewed the worksheet and reformatted it to 
make it easier to complete. 

DATA COLLECTION 

Twenty-six eMINTS instructional specialists recorded what they did during 2,367 
Classroom Visits to 355 teachers. They entered the information into the Classroom 
Visit Excel worksheets and uploaded these to the eMINTS Moodle. EDC/CCT 
evaluators downloaded these over the fall, winter, and spring of 2006-2007. After 
the final Classroom Visits had been made, EDC/CCT evaluators downloaded the final 
set of Classroom Visit records and entered the data into a Classroom Visit database. 
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TEACHER TECHNOLOGY LITERACY SURVEY 


INSTRUMENT DEVELOPMENT 

EDC/CCT evaluators collaborated with eMlNTS staff on the Teacher Technology 
Literacy Survey, which was administered online by the eMlNTS National Center to 
all program participants. eMlNTS staff developed the first eight questions, which 
asked for demographic information and for teachers to rate their level of 
understanding on a range of technology skills. They had used these questions in 
surveys they had conducted in previous years. EDC/CCT evaluators developed the 
next five questions, which asked about teachers’ technology access, their level of 
classroom use of a range of technological devices and applications, and their 
perceptions of challenges they might have experienced using technology in their 
teaching. Some of these questions were based on the eMlNTS requirements for 
teacher technology access stipulated on the eMlNTS program website. Other 
questions were derived from surveys of other technology professional development 
programs evaluated by EDC/CCT, such as the Intel® Teach Essential program and 
New Mexico's Regional Educational Technology Assistance program, and from 
surveys conducted for the Use, Support, and Effect of Instructional Technology 
(USEIT) Study conducted by the Technology and Assessment Study Collaborative 
( http://www.bc.edu/research/intasc/ ). 

DATA COLLECTION 

All eMlNTS participants were asked to complete the Teacher Technology Literacy 
Survey online in the spring of 2007. All 464 participants in Year 1 and Year 2 of 
eMlNTS were asked to complete the survey. Two hundred and thirty-one 
participants responded to the survey, for a response rate of 49.8%. 
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TEACHER MASTERY RUBRICS 


INSTRUMENT DEVELOPMENT 

To assess whether eMINTS participants understood the key concepts of the eMINTS 
program, EDC/CCT evaluators reviewed teacher portfolios. Participants who came 
from districts that funded their eMINTS participation through Title IID funds were 
required to submit teacher portfolios to receive eMINTS certification. In addition, 
teachers from some other districts also voluntarily submitted portfolios. eMINTS 
Comprehensive teachers had to submit a Lesson Plan, a WebQuest, and a Classroom 
Website as part of their portfolios, and the eMINTS4ALL participants had to submit 
a Lesson Plan as part of theirs. EDC/CCT evaluators created rubrics for evaluating all 
three of these artifacts. Because the portfolio artifacts are designed by teacher 
participants to guide their instruction or their interactions with students, the 
artifacts can serve as proxies, if not for teacher practice, then for teacher 
understanding of how to structure their instructional practice and use of technology 
to support students. 

The process of creating the Teacher Mastery rubrics was similar to the process used 
to create the fidelity instruments and consisted of the following steps: existing 
documentation review; initial rubric development; item review and validation; 
rubric training and assessment of inter-rater reliability; and final revision for use by 
program staff. 

EXISTING DOCUMENTATION REVIEW 

To create the Teacher Mastery rubrics, EDC/CCT evaluators first reviewed a number 
of existing documents to inform the design, in particular, the existing rubric for 
evaluating WebQuests; the constructivist Lesson Plan form; examples of Classroom 
Websites, WebQuests, and constructivist Lesson Plans; the hallmarks of an effective 
eMINTS classroom, and the eMINTS modules that focus on creating Classroom 
Websites, WebQuests, and Lesson Plans. These documents informed both the design 
of the rubrics and their content. 

INITIAL RUBRIC DEVELOPMENT 

Using the information derived from our review of existing documentation, we 
created three different rubrics: a Lesson Plan rubric, a Classroom Website rubric, 
and a WebQuest rubric. Evaluators aligned the Lesson Plan rubric format with the 
constructivist Lesson Plan form that eMINTS participants use. The WebQuest rubric 
was a revised version of the existing one, and the Classroom Website rubric was 
designed to address key content areas covered in the Classroom Website module. 

ITEM REVIEW AND VALIDATION 

As with the PD Fidelity instruments, we engaged in a process of gathering input 
from the eMINTS leadership team members to ensure the face and content validity 
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of the Teacher Mastery rubrics. Once the initial draft rubrics were created, 
evaluators shared them with eMINTS staff and held a series of meetings to go over 
the rubrics item by item to get their feedback on (a) whether the different items 
were appropriate, (b) whether all necessary content was covered, and (c) whether 
the staff agreed with the descriptions of what constituted high, medium, and low 
levels of mastery for each item. 

Expert feedback from the leadership team was used to revise the Teacher Mastery 
rubrics. These revised rubrics were then sent to eMINTS staff for a second round of 
review. Staff sent additional feedback, which was used to further revise the rubrics. 

RUBRIC TRAINING AND ASSESSMENT OF INTER-RATER RELIABILITY 

Three EDC/CCT evaluators were involved in scoring teacher portfolio artifacts. The 
evaluators were trained for a full day on the use of the rubrics. For each of the 
rubrics, evaluators were presented with a sample artifact. Evaluators went through 
each rubric, item by item, as they reviewed the teacher artifact and discussed how 
they would score it. Slight revisions to the rubrics were made during the trainings, 
particularly when language needed to be more precise. After training, the evaluators 
independently scored a common Lesson Plan, WebQuest, and Classroom Website. 

Three independent raters were required due to the volume of Lesson Plans, 
WebQuests, and Classroom Websites. Inter-rater reliabilities were taken throughout 
the scoring period to ensure that raters maintained high levels of agreement. The 
scores of the three raters were analyzed for inter-rater reliability using Cohen’s 
Kappa^. The goal was to achieve an average Kappa value that is considered 
substantial, 0.60 to 0.79, or outstanding, 0.80 and above, with the minimum 
acceptable Kappa being 0.50. Inter-rater reliability was performed on six Lesson 
Plans, four WebQuests, and four Classroom Websites. The average Kappa value for 
Lesson Plans ranged from 0.51 to 0.79. The average Kappa value for WebQuests 
ranged from 0.72 to 0.82, and the average Kappa value for Classroom Websites 
ranged from 0.64 to 0.84. These Kappa values were all significant at p < .01, showing 
sufficient inter-rater reliability. 

Three independent composite scores were then made by totaling each rubric item 
for Lesson Plans, WebQuests, and Classroom Websites. These composite scores 
constitute Teacher Mastery in each area. 

DATA COLLECTION 

The eMINTS instructional specialists collected portfolios from those teachers whose 
districts paid for their eMINTS professional development with Title IID funds, and 
from a few other districts that also had teachers volunteer to submit portfolios. 


^ Cohen’s Kappa coefficient is a statistical measure of inter-rater agreement that takes into account the 
likelihood of chance agreements and thus is a more valid measure than percent agreement calculations. 
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Along with the artifacts, these portfolios contained some demographic information 
about the teacher, such as the teacher’s school, district, and grade taught. In 
addition, the teachers needed to prove that they had actually used the Lesson Plan 
(all participants), WebQuest, and Classroom Website [eMINTS Comprehensive 
participants). Therefore, the portfolios needed to contain student work associated 
with these artifacts. 

The instructional specialists submitted the portfolios to the eMINTS National Center, 
after which they were sent to EDC/CCT. The portfolios were distributed to three 
EDC/CCT evaluators. Using the Teacher Mastery rubrics, these evaluators scored the 
teacher artifacts. One of the evaluators then recorded all of the scores in a Teacher 
Mastery database. Overall, EDC/CCT evaluators reviewed 180 Lesson Plans [99 
from eMINTS Comprehensive and 81 from eMINTS4ALL), 95 WebQuests, and 103 
Classroom Websites. The numbers for each artifact are slightly different due to 
some incomplete and corrupt files and sites upon data collection. 

RUBRIC REFINEMENT 

Once the scoring and the analysis of the Teacher Mastery scores were complete, 
EDC/CCT evaluators revisited the Teacher Mastery rubrics. Using frequency data 
from the analysis of Teacher Mastery scores — which showed the items that had 
little variation — and raters' own experience scoring dozens of artifacts, EDC/CCT 
evaluators edited the rubrics a final time, adding and subtracting items in some 
cases, but primarily revising the wording to make it even more precise and concrete. 
This was done to make the rubrics even more effective for future evaluations and 
research. These revised rubrics were then submitted to the eMINTS program staff 
for use in their own internal evaluation of the program, both in Missouri and in 
other states as the program is scaled up. 

STUDENT-LEVEL DEMOGRAPHIC AND MISSOURI ASSESSMENT 
PROGRAM (MAP) DATA 

DATA COLLECTION 

In order to measure the impact of the eMINTS program on students, EDC/CCT 
evaluators had to gain access to district and state student-level data. First, 
evaluators had to obtain information from the participating districts about the 
students and teachers in the schools that participated in the eMINTS program. We 
asked districts to complete rosters for these schools that provided student names 
and state ID numbers and the eMINTS status of their teachers [eMINTS veteran; 
eMINTS or eMINTS4ALL, Years 1 or 2; or non-eMINTS). These rosters gave us 
information on how many teachers were involved in each phase and program of 
eMINTS from each district. 


eMINTS Program Evaluation Report 


13 


The rosters then enabled us to collect student assessment data from students in 
each phase and program of eMlNTS. This gave a student sample that would allow us 
to examine the relationships among student performance, Program Fidelity, and 
Teacher Mastery. To accomplish this, we obtained the standardized test [the 
Missouri Assessment Program or MAP) data for students in the schools that 
received their eMINTS funding from Title IlD funds and from one other district that 
had teachers submit portfolios. EDC/CCT evaluators submitted a request for all 
Communication Arts [CA], Mathematics [MA), Science [SC), and Social Studies [SS) 
data from the specific schools to the Missouri Department of Elementary and 
Secondary Education, and they made these data available to EDC/CCT. In all, the 
MAP database contains data from 11,254 students in 40 schools from teachers that 
submitted portfolios [see Table 3). MAP data from students could not be used if it 
could not be matched to a roster from a participating, consent-giving teacher. 

Table 3: Number of students with at least one MAP test, by program type, 
with number of schools that those students attended 


Program Type 

Number of Students 

Number of Schools 

eMINTS Comprehensive 

2,119 

26 

eMINTS4ALL 

2,074 

22 

eMINTS Veterans 

152 

5 

non-eMINTS 

6,066 

21 

Program data unavailable 

155 

3 


TEACHER FOLLOW-UP INTERVIEWS 

EDC/CCT evaluators conducted interviews with 16 randomly selected teachers in 
the spring after they completed the eMINTS program. The interviews were designed 
to focus the discussion around a specific artifact, the Lesson Plan that teachers 
submitted as part of their eMINTS portfolios. The teachers were asked about some 
of the core eMINTS instructional strategies, what they learned from eMINTS about 
these strategies, and whether the specific Lesson Plan under discussion made use of 
these strategies. The interview data were analyzed using Atlas TI qualitative 
analysis software. 
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Section III: Analyses and database creation 

FREQUENCIES 

For each complete set of data we had, we tabulated basic frequencies, first to 
identify any problems with the data, so that the data could be cleaned, and then to 
see what the results were for each form of data collection. Preliminary and 
descriptive analyses were also run to ensure the data from all sources were 
compiled and cleaned accurately. Preliminary analyses also provided rich 
information about each variable from the instructional specialists, teachers, and 
students; they created a context in which to understand the subsequent, more 
advanced analyses. The findings from the frequency analyses of the PD Fidelity data, 
the Classroom Visit data, the Teacher Technology Literacy Survey, and the Teacher 
Mastery data are presented in Section IV. 

RELIABILITY AND FACTOR ANALYSES AND CREATION OF 
VARIABLES FOR ANALYSIS 

Each set of data had to be analyzed in a unique way, not only to make findings from 
that specific dataset meaningful, but also to prepare that dataset for integration into 
a comprehensive database that would allow for more complex analyses. In most 
cases, this meant that numerous data points had to be consolidated into a more 
comprehensive variable that could then be integrated into a larger database for 
complex analyses. For example, scores for the many different items on the Lesson 
Plan rubric had to be consolidated into a single cumulative Lesson Plan Mastery 
score. This section describes processes and analyses used for each dataset. 

PD FIDELITY 

In the process of creating the PD Fidelity instruments, EDC/CCT evaluators and 
eMlNTS staff worked to identify key constructs that eMlNTS program developers 
believed characterize a faithful, high-quality eMlNTS professional development 
experience, based on program theory of eMlNTS. The constructs identified were: 

• Program Logistics and Planning 

• Modeling Instruction 

• Community Building 

• Technology Utilization 

• Connection to Practice 

• Inquiry-Based Learning 
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When we created the Checklist and Snapshot instruments, we made sure to include 
multiple items that represented different aspects of each of these constructs. 

After the professional development observation data were collected, EDC/CCT 
evaluators conducted a reliability analysis [shown in Table 4), using Cronbach’s 
alpha^, on the items in each category of the Snapshot and Checklist instruments. 

This would demonstrate whether the items that we determined to be related fell 
reliably into one group or type of item. Before doing this analysis, we reviewed the 
frequencies for the Checklist items and removed any items that had very little 
variation (less than 20% or more than 80% of responses as ‘yes') from the reliability 
analysis. Eleven of 37 items were removed. The reliability analysis indicated the 
following reliability for the items that comprised the different constructs, with .6 
and higher being the standard cutoff for reliability: 


Table 4: Reliability of Checklist constructs 


Construct 

Reliability 

Program Logistics and Planning 

.384 

Modeling Instruction 

not enough items after reduction 

Community Building 

.517 

Technology Utilization 

.608 

Connection to Practice 

.478 

Inquiry-Based Learning 

.709 


It is important to remember that, because the Checklist was filled out only once per 
professional development session, the entire database includes only 50 data points 
per item. With few data points and only a single measure, with only 'yes’ or ‘no’ 
answers, there is low variability, and finding high reliability across only 4 to 7 items 
is unlikely, even with related items. Technology Utilization and Inquiry-Based 
Learning were the only two constructs that could be considered highly reliable for 
the Checklist. We did not use the Checklist data in any subsequent analyses. Despite 
the lack of reliability for some of the constructs, however, the item-by-items 
frequencies still provide useful information about the specific instructional 
techniques that instructional specialists are using in the workshops and the kind of 
instruction teacher participants are experiencing. 

We then engaged in the same process with the Snapshot data. Because the Snapshot 
instrument had observers record what was happening every 15 minutes, it included 
16 data points for every item, which created a larger dataset to work with than the 
Checklist. The Snapshot items were summed and averaged across the number of 


Cronbach’s alpha is a measure of reliability that varies between 0 and 1 and increases as the correlations 
between the items increase. 
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Snapshots taken. For example, if item 1 was marked 'yes' on 8 out of 10 Snapshots 
taken by an observer, that instructional specialist received 0.8 for that item. This 
allowed for much more variability than the O’s and I’s utilized by the one-time 
Checklist. 

We first removed the items with low variability [since there was more variability, 
only items with less than 15% or more than 85% of responses as 'yes']. Thirteen of 
44 items were removed. We then ran a reliability analysis on the remaining items in 
each category, [shown in Table 5), with .6 again indicating a reliable construct. 


Table 5: Reliability of Snapshot theory-based constructs 


Construct 

Reliability 

Program Logistics and Planning 

not enough items after reduction 

Modeling Instruction 

.405 

Community Building 

.707 

Technology Utilization 

.600 

Connection to Practice 

.689 

Inquiry-Based Learning 

.784 


The Snapshot reliability analysis determined that four of the six theory-based 
constructs created by eMlNTS program staff and EDC/CCT evaluators were 
statistically reliable. These four factors could then reliably represent the constructs 
they were intended to represent in further analyses that look at the relationship 
between PD Fidelity and Teacher Mastery and student outcomes. Although 
Modeling Instruction did not have high internal reliability as a construct, this is 
probably because, once items with low variation were removed, the construct 
comprised only four items. It does not mean that the construct is not meaningful. 
Therefore, we still included the factor in later analyses because the items that 
comprise Modeling Instruction are important aspects of the professional 
development and have face validity. An overall fidelity measure. Program Construct 
Fidelity, was computed by averaging all of the composite scores for all of the original 
theory-based snapshot constructs. 

The following boxes list theory-based constructs and the individual Snapshot items 
comprising them. Item numbers indicate placement on the original instrument. 
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Modeling Instruction 


1. Facilitator is presenting session content 

2. Facilitator is introducing, giving directions for, or summing up an activity 

3. Facilitator is working with groups of participants/individual participants on tasks and activities 
5. Facilitator is asking open-ended questions 


Community Building 

17. Facilitator is encouraging participants to answer each other's questions 

36. Participants are asking questions of peers/facilitator 

37. Participants are answering each other's questions 

38. Participants are working independently 

39. Participants are sharing ideas, resources 

40. Participants are working in cooperative groups 

41. Participants are engaging in community building 


Technology Utilization 

24. Facilitator is modeling ways to communicate information with technology 

25. Facilitator is modeling use of eMINTS resources as instructional tools 

27. Facilitator is using technology (e.g., SMART Board™) to facilitate discussions or present session materials 

42. Participants are using technology for activities or research 

43. Participants are using technology for presentations or to create materials 


Connection to Practice 

8. Facilitator is encouraging reflection on teaching practices and experiences 

9. Facilitator is connecting session topic to practice 

11. Facilitator is using examples from his/her observations of participants' classroom teaching 

12. Facilitator is providing examples of assessment tools. Lesson Plans, activities, or resources for classroom use 

29. Participants are connecting session topic to their practice 

30. Participants are engaged in an activity or working on a lesson related to content of session 

31. Participants are reflecting on what they are learning in the session 


Inquiry-Based Learning 

13. Facilitator is engaging participants in activities that enable them to formulate questions to drive 
lesson/activity 

14. Facilitator is engaging participants in activities that enable them to investigate their questions 
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15. Facilitator is presenting resources and strategies for effective exploration and research 

16. Facilitator is engaging participants in activities that enable them to analyze or synthesize what they have 
found through their research 

32. Participants are developing their own questions to investigate 

33. Participants are gathering information from a variety of resources 

34. Participants are analyzing or synthesizing information that they have gathered 

35. Participants are presenting information 

We then ran an exploratory factor analysis on the Snapshot items to identify 
whether another set of factors representing PD Fidelity could be found in the data. 
Again, we removed 13 out of 44 items because of low variability. Principal axis 
factoring^ was performed using oblimin rotation^ on all items. The factor analysis 
produced five groupings of related items. EDC/CCT evaluators reviewed each group 
of items. Table 6 presents the five factor names given to the item sets, followed by 
their reliability levels. 

Table 6: Reliability for Snapshot Factor Analysis-Based Constructs 


Factors 

Reliability 

Structured Activities 

.530 

Participant-Led Discussion 

.715 

Scaffolding Instruction 

.828 

Facilitating Discussion 

.823 

Active Work/Learning 

.795 


The following boxes list theory-based constructs and the individual Snapshot items 
comprising them. Item numbers indicate placement on the original instrument. 

structured Activities 

11. Facilitator is using examples from his/her observations of participants' classroom teaching 

31. Participants are reflecting on what they are learning in the session 

35. Participants are presenting information 

40. Participants are working in cooperative groups 


^ Principal axis factoring is a type of principal components analysis that looks for a combination of items 
that explains the maximum proportion of the variance of those items. This leads to factors that have the 
strongest reliability. 

® Oblimin rotation is the standard method of factoring when one wishes to find a solution that allows the 
factors to be correlated (a non-orthogonal solution). This allows the analysis to consider all possible 
combinations of items in determining factors. 
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Participant-Led Discussion 

36. Participants are asking questions of peers/facilitator 
39. Participants are sharing ideas, resources 

41. Participants are engaging in community building 

37. Participants are answering each other's questions 

38. Participants are working independently 


Scaffolding Instruction 

15. Facilitator is presenting resources and strategies for effective exploration and research 

27. Facilitator is using technology (e.g., SMART Board™) to facilitate discussions or present session materials 

1. Facilitator is presenting session content 

9. Facilitator is connecting session topic to practice 

24. Facilitator is modeling ways to communicate information with technology 

2. Facilitator is introducing, giving directions for, or summing up an activity 

12. Facilitator is providing examples of assessment tools. Lesson Plans, activities, or resources for classroom use 


Facilitating Discussion 

5. Facilitator is asking open-ended questions 

8. Facilitator is encouraging reflection on teaching practices and experiences 
17. Facilitator is encouraging participants to answer each other's questions 
29. Participants are connecting session topic to their practice 


Active Work/Learning 

3. Facilitator is working with groups of participants/individual participants on tasks and activities 

13. Facilitator is engaging participants in activities that enable them to formulate questions to drive 
lesson/activity 

16. Facilitator is engaging participants in activities that enable them to analyze or synthesize what they have 
found through their research 

30. Participants are engaged in an activity or working on a lesson related to content of session 

32. Participants are developing their own questions to investigate 

33. Participants are gathering information from a variety of resources 

34. Participants are analyzing or synthesizing information that they have gathered 

42. Participants are using technology for activities or research 

43. Participants are using technology for presentations or to create materials 
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An overall fidelity measure, called Factor Analysis-Based Fidelity (FA-Based 
Fidelity), was computed by averaging all of the composite scores for these factors. 
This overall fidelity measure and the individual factors are used in further analyses 
of Teacher Mastery, Classroom Visits, and student impact. 

CLASSROOM VISITS 

Instructional specialists visited their teacher participants up to 10 times over the 
school year and recorded in a spreadsheet the amount of time [minutes) they spent 
assisting teachers in each of the following areas: Modeling Instruction, Lesson 
Planning, Technology Assistance, Reflective Practice, Problem Solving, and other. 

In order to compare across all cases the kind of support received by teachers from 
their instructional specialists, EDC/CCT evaluators summed for each teacher the 
number of minutes spent in each area across visits and then divided by the total 
number of minutes spent in visitation, to obtain the percentage of time the 
instructional specialist spent assisting the teacher in a particular area.^ The number 
of visits made to each teacher by the instructional specialist provided another 
variable for analysis. 

TEACHER TECHNOLOGY LITERACY SURVEY 

To prepare the Teacher Technology Literacy Survey data to be included in the 
combined analyses of all the datasets, EDC/CCT evaluators made composite 
variables out of sets of items. For example one question presented respondents with 
statements about challenges they may have encountered in using technology in their 
teaching and asked them to rate their level of agreement. Certain questions were 
reverse-coded because the statements were positively and negatively phrased. 
Factor analysis^ was performed on all of the items in that question, to determine if 
certain challenges were related to each other. The analysis indicated that two sets of 
items grouped together. One set included items associated with technology access, 
such as, 'An adequate number of computers were available,' and another set were 
associated with support issues, such as, 'I have had adequate technical support’ 

Thus scores were obtained for computer challenge issues and support challenge 
issues by taking the average of the responses for each, respectively. 

The two variables created from the Teacher Technology Literacy Survey, here on 
referred to as Computer Issues and Support Issues, were used in further analyses 
discussed below. 


’ For example, if 415 minutes across the visits were spent in helping the teacher model instruction and the 
total number of minutes in visitation was 2502, then the percentage of time spent in that area was 16.6%. 

* Specifically, principal axis factoring with oblimin rotation^’^. 
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TEACHER MASTERY 

EDC evaluators computed Teacher Mastery scores for each artifact (Lesson Plan, 
WebQuest, Classroom Website) by totaling the scores from each rubric for every 
teacher. The Lesson Plan rubric had a total of 21 items, each with a low score of 1, a 
medium score of 2, and a high score of 3, for a range of 21-63 points. Both the 
WebQuest and Classroom Website rubrics had a total of 17 items each with the 
same 1-3 scoring system, for a range of 17-51 for each of these artifacts. The overall 
Teacher Mastery scores were then used in further analyses described below. 
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CREATION OF THE COMPREHENSIVE EMINTS DATABASE 


BUILDING THE FULL TEACHER DATABASE 

The full teacher database was huilt by merging data from four databases. The 
Classroom Visit (CV) database was chosen as the starting database into which the 
other three sets — Teacher Mastery [TM), Teacher Survey (TS), and PD Fidelity 
Snapshot [SS) — would be merged, as it contained the most teacher-level data. 

To begin, we decided to merge the TS data into the CV database. Within the survey 
database, one dummy-coded variable stood for both years (1, 2) and both eMINTS 
programs (4ALL, Comp PD). That variable was parsed out into two separate year 
and program variables using syntax. The TS data were then merged into the CV 
database using menu commands and matching by teacher code. Teacher names, 
districts, year, and program were double-checked to ensure the merge was 
successful. This merged database will now be referred to as the full database. 

The TM database did not yet include teacher codes, so we merged the teacher code 
variable from the full database by matching last and first names of teachers. As a 
number of teachers had TM data but no CV or TS data, some teacher names were left 
over with no codes attached. The missing codes were filled in by matching teachers 
and codes from a prior database. Since the program variable in the TM database was 
dummy-coded, this coding was transformed to "4ALL” or "Comp PD.” The TM 
database was then merged into the full database by matching teacher code. The 
names of instructional specialists, teachers, and districts were double-checked to 
make sure the merge was successful. 

The SS database was the last to be merged. Year and program variables were 
rewritten to the standardized form included in the other databases ["4ALL” and 
"Comp PD"). Instructional specialist names were also not standardized, so that 
variable was rewritten to match the instructional specialist name form in the other 
databases as well. Upon doing this, we found that there were some teachers in the 
full database that did not have an instructional specialist name attached — these 
were teachers who had TS data and not CV data. To rectify the missing instructional 
specialist names, we consulted the spreadsheet that listed the teacher assignment to 
each instructional specialist. We double-checked teacher assignment to 
instructional specialist for each teacher listed in the database. 

In the SS database, some instructional specialists had multiple observations. These 
scores were simply averaged together so each instructional specialist or pair of 
instructional specialists had one overall fidelity score. If an instructional specialist 
had multiple scores, but they were for different types of program (e.g., one 
observation for eMINTS Comp PD and one for eMINTS4ALL; or one for Year 1 
teachers and one for Year 2 teachers), then those were left separate. Teachers in the 
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instructional specialist’s complete scores would be matched with the fidelity 
observation [s) from that instructional specialist’s complete observation, and vice 
versa. The SS database was then imported, matching by instructional specialist, 
year, and type of eMlNTS program, so that each teacher was paired with the 
appropriate fidelity scores from his or her instructional specialist. 

The overall database was then checked to ensure that all importing was successful 
and all teacher and instructional specialist data corresponded appropriately. All 
variables were standardized [e.g., ensuring school names were spelled the same 
each time) so that statistics could be cleanly run. Once the database was complete, 
all teacher and instructional specialist names were removed and replaced with 
codes, preserving confidentiality. 

BUILDING THE FULL STUDENT DATABASE 

The full student database was built by merging teacher and instructional specialist 
information into a database with all student MAP scores. Students with at least one 
MAP score [out of Communication Arts, Mathematics, Science, and Social Studies) 
from grades 3 through 8 were included in the database. Only students on the rosters 
we received from the participating districts were included. 

In the student database, students were matched with their teacher’s code and the 
teacher’s instructional specialist’s code. All teacher and instructional specialist 
information was imported [via matching the teacher and instructional specialist 
codes) to align with each student, so that statistics could be run to determine any 
relationship between teacher and instructional specialist variables and MAP scores. 
Once the database was assembled, it was cleaned and checked for importing 
accuracy. Student names were removed and replaced with codes to preserve 
confidentiality. 

ANALYSIS OF THE FULL TEACHER DATABASE AND FULL STUDENT 
DATABASE 

Descriptive and preliminary analyses were run first to determine the variability and 
basic statistics of each measure and to lay a contextual foundation for more- 
advanced analyses. Descriptive statistics were utilized for the following: 

• Analyzing the Checklist items 

• Analyzing the Snapshot items 

• Analyzing what teachers and instructional specialists are spending their time on 
during the Classroom Visits, as well as how many visits are taking place, and 
whether any of this varies hy year, program, or eMINTS staff vs. PD4ETS 

• Analyzing Teacher Mastery artifacts and individual items on each rubric 
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• Looking at program participation across districts 

Correlations were run among teacher variables to determine relationships within 
the teacher level. They were also run to determine relationships across teacher, 
instructional specialist, and student variables. Specifically, correlations were run to 
analyze relationships between: 

• PD Fidelity and Teacher Mastery 

• PD Fidelity and the Teacher Technology Literacy Survey 

• Teacher Mastery and Classroom Visits 

• Classroom Visits and PD Fidelity 

• Program and teacher findings and student MAP scores 

Analysis-of-variance tests were also run to analyze differences between eMlNTS 
staff and external instructional specialists and to look at differences in outcome 
measures across program year and type. These tests were run for: 

• Comparing the Checklist items for eMINTS staff vs. PD4ETS 

• Comparing the Snapshot items for eMINTS staff vs. PD4ETS 

• Comparing the Classroom Visits across program, year, and eMINTS staff vs. PD4ETS 

• Examining covariates of student MAP scores 

• Analyzing the relationship between Teacher Mastery and student MAP scores 

• Analyzing the relationship between PD Fidelity and student MAP scores 

• Analyzing the relationship between Classroom Visits and student MAP scores 

Lastly, where appropriate, regression analyses were run to predict student 
outcomes while controlling for appropriate student covariates. Analysis was run to: 

• Analyze the relationship between PD Fidelity and Teacher Mastery 

• Analyze the relationship between program and teacher findings and student MAP 
scores 
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Section IV: Findings 

PD FIDELITY 

The first step in the process of this multi-layered exploration of the eMlNTS 
program was to understand whether the program is being implemented with 
fidelity. The two observation instruments were designed to capture different kinds 
of information about PD Fidelity. The Checklist, which is completed only once during 
a professional development session, contains items that program designers would 
expect to see at least once during a faithful, high-quality eMINTS professional 
development session. 

Table 7 shows the percentage of times across all professional development sessions 
that each item occurred. For example, the second item down, "Consistently engaged 
all participants," occurred in 92.0% of all sessions observed. The information in the 
table is organized by construct, with each item listed that loads onto that construct. 

Table 7: Percentage of professional development sessions in which key 
activities from the Checklist occurred 


Construct 

Item 

Percent (%) 

Modeling Instruction 

3. Presented the sessions' essential question/goals at 
the beginning of the session 

84.0 


5. Consistently engaged all participants 

92.0 


10. Reviewed session goals and/or essential question 
and tied it to the session's content 

60.0 

Technology Utilization 

4. Focused primarily on technology and not on its 
integration into teaching 

6.0 


7. Used own technology-integrated lessons/artifacts 

48.0 


16. Used technology to engage in 
lessons/activities/presentations 

88.0 


23. Discussed online tools and resources that support 
instructional practice 

66.0 


24. Discussed challenges of integrating technology 
into classroom practice 

58.0 


26. Discussed how technology can be used to support 
community building 

12.0 


30. Discussed how technology can be used to support 
inquiry-based learning 

52.0 
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Community Building 

8. Was respectful toward participants 

100.0 


9. Used deliberate strategies to group participants 

46.0 


17. Engaged in collaborative group work 

86.0 


18. Engaged in whole-group discussion 

92.0 


25. Discussed techniques for community building 

18.0 


26. Discussed how technology can be used to support 
community building 

12.0 


27. Discussed techniques for collaborative learning 

22.0 


28. Discussed the challenges of classroom use of 
collaborative learning techniques 

16.0 


29. Discussed meeting the needs of diverse learners 

40.0 

Connection to Practice 

6. Allowed time for participants to discuss, reflect, 
and share 

98.0 


11. Debriefed participants on the session's activities 
and their implementation in the classroom 

66.0 


15. Engaged in or created a lesson/activity that could 
be used with students 

86.0 


22. Reflected on what they learned in the session 

80.0 


34. Discussed the integration of higher-order thinking 
skills in lesson and activity planning 

46.0 


35. Discussed alternative assessment strategies 
(rubrics, portfolios, performances-based assessment) 

26.0 


36. Discussed the role of assessment in shaping 
instruction 

24.0 


37. Discussed how to apply what they learned in the 
session to their own practice 

90.0 

Inquiry-Based Learning 

19. Developed questions to drive 
exploration/research 

52.0 


20. Engaged in exploration/research 

68.0 


21. Analyzed and synthesized what they learned 

80.0 


30. Discussed how technology can be used to support 

52.0 


31. Discussed techniques for IBL inquiry-based 
learning 

32.0 


32. Discussed the challenges of integrating inquiry- 
based learning into classroom practice 

32.0 


33. Discussed how session topic can support inquiry- 
based learning 

44.0 
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Program Logistics and 
Planning 

1. Provided a session agenda (online or paper) 

92.0 


2. Verifying that equipment is fully functioning 

88.0 


12. Arranged for meals/food 

86.0 


13. Ended the session on time 

86.0 


14. Returned facility to its original state 

86.0 


These findings for the Checklist data indicate that, for many of the key eMINTS 
instructional strategies — such as engaging "in collaborative group work," having 
"time for participants to reflect, discuss, and share," engaging in or creating "a 
lesson/activity that can be used with students," and connecting "the session topic to 
participants' practice" — there is a high level of fidelity across the professional 
development sessions. However, some instructional approaches, particularly the 
explicit discussions around Community Building and Inquiry-Based Learning, are 
not occurring in the majority of sessions. The Checklist frequencies suggest that 
eMINTS instructional specialists are, overall, well-prepared to model the kind of 
instruction eMINTS advocates, but may need extra support and professional 
development in engaging participants in explicit discussions about some of the 
instructional strategies and techniques that are integral to the eMINTS approach to 
teaching and learning. 

We also examined the Checklist data by whether the instructional specialist was an 
eMINTS staff member or was a participant or graduate of the PD4ETS program [see 
Table 8). We found that were no significant differences between the PD4ETS 
participants and graduates and the eMINTS staff on any of the Checklist items 
except one, "used deliberate strategies to group participant." eMINTS staff engaged 
in this practice significantly more than their PD4ETS counterparts. 


Table 8: Proportion of professional development sessions in which key 
activities from the Checklist occurred, by staff and PD4ETS 


Item 

Staff 

PD4EST 


Mean (SD) 

Mean (SD) 

1. Provided a session agenda (online or paper) 

.86 (.35) 

.96 (.19) 

2. Verified that equipment is fully functioning 

.91 (.29) 

.86 (.36) 

3. Presented the sessions' essential question/goals at the beginning of the 
session 

.86 (.35) 

.82 (.39) 

4. Focused primarily on technology and not on its integration into teaching 

.09 (.29) 

.04 (.19) 

5. Consistently engaged all participants 

.91 (.29) 

.93 (.26) 
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6. Allowed time for participants to discuss, reflect, and share 

1.00 (.00) 

.96 (.19) 

7. Used own technology-integrated lessons/artifacts 

.55 (.51) 

.43 (.50) 

8. Was respectful toward participants 

1.00 (.00) 

1.00 (.00) 

9. Used deliberate strategies to group participants 

.68 (.48) 

.29 (.46) 

10. Reviewed session goals and/or essential question and tied it to the 
session's content 

.68 (.48) 

.54 (.51) 

11. Debriefed participants on the session's activities and their 
implementation in the classroom 

.64 (.49) 

.68 (.48) 

12. Arranged for meals/food 

.86 (.35) 

.86 (.36) 

13. Ended the session on time 

.86 (.35) 

.86 (.36) 

14. Returned facility to its original state 

.86 (.35) 

.86 (.36) 

15. Engaged in or created a lesson/activity that could be used with students 

.91 (.29) 

.82 (.39) 

16. Used technology to engage in lessons/activities/presentations 

.91 (.29) 

.86 (.36) 

17. Engaged in collaborative group work 

.96 (.21) 

.79 (.42) 

18. Engaged in whole-group discussion 

.91 (.29) 

.93 (.26) 

19. Developed questions to drive exploration/research 

.55 (.51) 

.50 (.51) 

20. Engaged in exploration/research 

.72 (.46) 

.64 (.49) 

21. Analyzed and synthesized what they learned 

.86 (.35) 

.75 (.44) 

22. Reflected on what they learned in the session 

.77 (.43) 

.82 (.39) 

23. Discussed online tools and resources that support instructional practice 

.59 (.50) 

.71 (.46) 

24. Discussed challenges of integrating technology into classroom practice 

.55 (.51) 

.61 (.50) 

25. Discussed techniques for community building 

.27 (.46) 

.11 (.32) 

26. Discussed how technology can be used to support community building 

.14 (.35) 

.11 (.32) 

27. Discussed techniques for collaborative learning 

.23 (.43) 

.21 (.42) 

28. Discussed the challenges of classroom use of collaborative learning 
techniques 

.23 (.43) 

.11 (.32) 

29. Discussed meeting the needs of diverse learners 

.36 (.49) 

.43 (.51) 

30. Discussed how technology can be used to support inquiry-based 
learning 

.46 (.51) 

.57 (.51) 

31. Discussed techniques for inquiry-based learning 

.32 (.48) 

.32 (.48) 

32. Discussed the challenges of integrating inquiry-based learning into 
classroom practice 

.32 (.48) 

.32 (.48) 

33. Discussed how session topic can support inquiry-based learning 

.50 (.51) 

.40 (.50) 

34. Discussed the integration of higher-order thinking skills in lesson and 
activity planning 

.41 (.50) 

.50 (.51) 

35. Discussed alternative assessment strategies (rubrics, portfolios, 
performances-based assessment) 

.23 (.43) 

.29 (.46) 
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36. Discussed the role of assessment in shaping instruction 

.27 (.46) 

.21 (.42) 

37. Discussed how to apply what they learned in the session to their own 
practice 

.82 (.40) 

.96 (.19) 


The Snapshot instrument was designed to capture information not only about what 
happened during a professional development session, but also about how often 
certain things happened. The Snapshot instrument, completed up to 16 times in a 
four-hour session, contains items that program designers would expect to see 
frequently during a faithful, high-quality eMINTS professional development session. 

Table 9 presents data showing the frequency with which the activities described in 
the Snapshot occurred during professional development sessions. For each 
instructional specialist, the number of times each Snapshot item was checked off 
(observed in a session) was divided by the total number of Snapshots taken during a 
session, resulting in the proportion of time the item was present across Snapshots. 
The minimum proportion of time is listed, along with the maximum proportion of 
time. The proportions for each instructional specialist were also averaged, shown in 
the "Mean" column with the standard deviation in parenthesis. 

For example, the eighth item down, "Encouraging reflection on teaching practices 
and experiences”, occurred in a professional development session at a minimum 0% 
of the time and at a maximum 45% of the time. The average proportion of time the 
item occurred across Snapshots was 17%, with a standard deviation of 15%. 


Table 9: Proportion of Snapshots that recorded various activities as 
occurring 


Item 

Min 

Max 

Mean 

(SD) 

1. Presenting session content 

.00 

.50 

.20 (.13) 

2. Introducing, giving directions for, or summing up an activity 

.07 

.55 

.30 (.14) 

3. Working with groups of participants/individual participants on tasks and 
activities 

.08 

.70 

.41 (.18) 

4. Connecting session topic to other eMINTS training sessions 

.00 

.08 

.02 (.03) 

5. Asking open-ended questions 

.00 

.46 

.23 (.15) 

6. Asking questions to which he/she knows the answer 

.00 

.17 

.02 (.05) 

7. Providing a "brain break" for participants 

.00 

.23 

.03 (.06) 

8. Encouraging reflection on teaching practices and experiences 

.00 

.45 

.17 (.15) 

9. Connecting session topic to practice 

.00 

.82 

.22 (.19) 

10. Connecting session topic and educational standards 

.00 

.08 

.005 (.02) 

11. Using examples from observations of participants' classroom teaching 

.00 

.09 

.02 (.04) 
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12. Providing examples of assessment tools, Lesson Plans, activities, or 
resources for classroom use 

.00 

.36 

.12 (.12) 

13. Engaging participants in activities that enable them to formulate questions 
to drive lesson/activity 

.00 

.27 

.09 (.10) 

14. Engaging participants in activities that enable them to investigate their 
questions 

.00 

.45 

.07 (.13) 

15. Presenting resources and strategies for effective exploration and research 

.00 

.27 

.04 (.08) 

16. Engaging participants in activities that enable them to analyze or 
synthesize what they have found through their research 

.00 

.45 

.08 (.13) 

17. Encouraging participants to answer each other's questions 

.00 

.45 

.13 (.14) 

18. Limiting participant discussion or input 

.00 

.07 

.005 (.02) 

19. Facilitating the sharing of ideas and experiences 

.00 

.55 

.25 (.14) 

20. Discussing effective techniques of classroom community building 

.00 

.09 

.006 (.02) 

21. Ignoring conflicts among group members 

.00 

.00 

.00 (.00) 

22. Allowing an individual(s) to dominate conversation or intimidate others 

.00 

.36 

.02 (.09) 

23. Showing frustration with technology 

.00 

.00 

.00 (.00) 

24. Modeling ways to communicate information with technology 

.00 

.21 

.08 (.07) 

25. Modeling use of eMINTS resources as instructional tools 

.00 

.29 

.06 (.09) 

26. Presenting technology irrelevant to session's content and/or to classroom 
utilization (other than a "brain break") 

.00 

.07 

.005 (.02) 

27. Using technology (e.g., SMART Board™) to facilitate discussions or 
present session materials 

.00 

.82 

.29 (.22) 

28. Participants are expressing their confusion about a task/activity they are 
asked to do. 

.00 

.14 

.03 (.05) 

29. Participants are connecting session topic to their practice. 

.00 

.73 

.27 (.19) 

30. Participants are engaged in an activity or working on a lesson related to 
content of session. 

.00 

1.0 

.50 (.28) 

31. Participants are reflecting on what they are learning in the session. 

.00 

.46 

.14 (.11) 

32. Participants are developing their own questions to investigate. 

.00 

.15 

.02 (.05) 

33. Participants are gathering information from a variety of resources. 

.00 

.40 

.09 (.11) 

34. Participants are analyzing or synthesizing information they have gathered. 

.00 

.55 

.14 (.15) 

35. Participants are presenting information. 

.00 

.27 

.09 (.08) 

36. Participants are asking questions of peers/facilitator. 

.07 

.64 

.40 (.17) 

37. Participants are working independently. 

.00 

.75 

.32 (.21) 

38. Participants are answering each other's questions. 

.00 

.58 

.32 (.18) 

39. Participants are sharing ideas, resources. 

.00 

.67 

.31 (.20) 

40. Participants are working in cooperative groups. 

.00 

.42 

.20 (.15) 
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41. Participants are engaging in community building. 

.00 

.14 

.05 (.06) 

42. Participants are using technology for activities or research. 

.00 

.85 

.31 (.25) 

43. Participants are using technology for presentations or to create materials. 

.00 

.38 

.13 (.14) 

44. Participants are sharing non-educational technological resources or tools. 

.00 

.07 

.005 (.02) 


Because the Snapshot instruments record what happens throughout the entire 
professional development session, and because the eMINTS professional 
development sessions have many different kinds of activities happening over the 
four hours, no single activity is likely to occur a majority of the time. In the 
frequencies for the Checklist, percentages in the 80’s and 90’s indicate a high level of 
fidelity across observations because the Checklist presents binary findings — the 
activity either happened or did not. The Snapshot frequencies show not only 
whether something happened but how often. Each item is more meaningful when 
looked at in relation to the other items. For example, the fact that the instructional 
specialists spent 20% of the time presenting session content is more meaningful 
when contrasted with the finding that instructional specialists were working with 
groups of participants on activities 41% of the time, that participants were working 
on an activity or lesson 50% of the time, and that participants were presenting 
information 27% of the time. These four statistics viewed together present an 
interesting picture of eMINTS professional development sessions. One-fifth of the 
time, on average, instructional specialists provided content information to 
participants, and they spent twice as much time working with participants in small 
groups. Participants spent half of their time in the sessions engaged in a lesson or 
activity; during most of that time the instructional specialists were working with 
them, and a quarter of the time they were presenting information themselves. This 
balance between instructional specialist and participant presentations and 
engagement by participants in facilitated activities, is consistent with the eMINTS 
approach to professional development and instruction. 

These frequencies show that many other of the core components of the eMINTS 
model were being implemented regularly. Forty percent of the time participants 
were asking questions of their peers or the facilitator, and 32% of the time they 
were answering each other's questions, which suggests that a lot of back and forth 
discussion was taking place, rather than lecturing. In addition, frequencies were 
very low on the negative items, such as "limiting participants' discussion or input” 
or "showing frustration with technology.” However, the Snapshot frequency data 
suggest that instructional specialists were spending less time on some key 
instructional practices, such as Community Building and using the techniques of 
Inquiry-Based Learning, like having participants develop their own questions, 
gather information, and analyze and synthesize that information. 
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We also compared the level of fidelity, as determined by the Snapshot data, for 
eMINTS staff and for participants and graduates of the PD4ETS program, using a 
multivariate analysis of variance^. Table 10 presents the item-by-item Snapshot 
means and standard deviations for staff and for PD4ETS participants and graduates. 
The F statistic shows whether there was a significant difference between the two 
groups on how often that particular item was present. These findings show that, on 
most Snapshot items, there were no significant differences between staff and 
PD4ETS participants and graduates. Where they do differ, PD4ETS participants and 
graduates often engaged in those practices associated with the eMINTS model with 
greater frequency than staff, indicating that the PD4ETS program is preparing them 
to deliver the eMINTS professional development with a high level of fidelity. 


Table 10: Proportion of Snapshots that recorded various activities as 
occurring, by staff and PD4ETS 


Item 

Staff 

PD4ETS 



Mean (SD) 

Mean (SD) 

F 

1. Presenting session content 

.14 (.12) 

.22 (.15) 

4.07** 

2. Introducing, giving directions for, or summing up an activity 

.25 (.16) 

.29 (.12) 

ns 

3. Working with groups of participants/individual participants on 
tasks and activities 

.32 (.21) 

.43 (.22) 

3.38* 

4. Connecting session topic to other eMINTS training sessions 

.02 (.03) 

.04 (.06) 

ns 

5. Asking open-ended questions 

.21 (.15) 

.22 (.13) 

ns 

6. Asking questions to which he/she knows the answer 

.03 (.05) 

.01 (.04) 

2.98* 

7. Providing a "brain break" for participants 

.02 (.04) 

.02 (.06) 

ns 

8. Encouraging reflection on teaching practices and experiences 

.13 (.14) 

.14 (.12) 

ns 

9. Connecting session topic to practice 

.18 (.18) 

.24 (.16) 

ns 

10. Connecting session topic and educational standards 

.02 (.05) 

.01 (.02) 

ns 

11. Using examples from his/her observations of participants' 
classroom teaching 

.05 (.07) 

.02 (.06) 

ns 

12. Providing examples of assessment tools. Lesson Plans, 
activities, or resources for classroom use 

.08 (.10) 

.10 (.10) 

ns 

13. Engaging participants in activities that enable them to 
formulate questions to drive lesson/activity 

.04 (.07) 

.11 (.11) 

6.33** 

14. Engaging participants in activities that enable them to 
investigate their questions 

.05 (.11) 

.07 (.11) 

ns 

15. Presenting resources and strategies for effective exploration 
and research 

.02 (.06) 

.04 (.09) 

ns 


^ Multivariate analysis of variance is a statistical test that allows for comparisons between one or more 
groups across several different measures. 
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16. Engaging participants in activities that enable them to 
analyze or synthesize what they have found through their 
research 

.06 (.11) 

.07 (.10) 

ns 

17. Encouraging participants to answer each other's questions 

.09 (.12) 

.11 (.11) 

ns 

18. Limiting participant discussion or input 

.00 (.02) 

.00 (.01) 

ns 

19. Facilitating the sharing of ideas and experiences 

.23 (.14) 

.25 (.11) 

ns 

20. Discussing effective techniques of classroom community 
building 

.02 (.04) 

.01 (.02) 

ns 

21. Ignoring conflicts among group members 

.00 (.00) 

.00 (.00) 

ns 

22. Allowing an individual(s) to dominate conversation or 
intimidate others 

.00 (.00) 

.01 (.07) 

ns 

23. Showing frustration with technology 

.01 (.02) 

.01 (.06) 

ns 

24. Modeling ways to communicate information with 
technology 

.05 (.06) 

.08 (.09) 

ns 

25. Modeling use of eMINTS resources as instructional tools 

.04 (.06) 

.04 (.06) 

ns 

26. Presenting technology irrelevant to session's content and/or 
to classroom utilization (other than a "brain break") 

.01 (.03) 

.00 (.02) 

ns 

27. Using technology (e.g., SMART Board™) to facilitate 
discussions or present session materials 

.17 (.20) 

.26 (.22) 

ns 

28. Participants are expressing their confusion about a 
task/activity they are asked to do. 

.02 (.04) 

.02 (.05) 

ns 

29. Participants are connecting session topic to their practice. 

.23 (.17) 

.30 (.16) 

ns 

30. Participants are engaged in an activity or working on a 
lesson related to content of session. 

.33 (.26) 

.51 (.29) 

4.91** 

31. Participants are reflecting on what they are learning in the 
session. 

.14 (.16) 

.12 (.10) 

ns 

32. Participants are developing their own questions to 
investigate. 

.04 (.08) 

.04 (.08) 

ns 

33. Participants are gathering information from a variety of 
resources. 

.10 (.11) 

.10 (.12) 

ns 

34. Participants are analyzing or synthesizing information they 
have gathered. 

.08 (.13) 

.12 (.11) 

ns 

35. Participants are presenting information. 

.10 (.11) 

.08 (.09) 

ns 

36. Participants are asking questions of peers/facilitator. 

.35 (.21) 

.43 (.15) 

2.81* 

37. Participants are working independently. 

.32 (.23) 

.37 (.21) 

ns 

38. Participants are answering each other's questions. 

.29 (.22) 

.37 (.19) 

ns 

39. Participants are sharing ideas, resources. 

.34 (.22) 

.40 (.20) 

ns 

40. Participants are working in cooperative groups. 

.24 (.16) 

.22 (.20) 

ns 

41. Participants are engaging in community building. 

.06 (.07) 

.06 (.09) 

ns 


eMINTS Program Evaluation Report 


34 


42. Participants are using technology for activities or research. 

.19 (.18) 

.32 (.22) 

5.20** 

43. Participants are using technology for presentations or to 
create materials. 

.18 (.15) 

.14 (.17) 

ns 

44. Participants are sharing non-educational technological 
resources or tools. 

.02 (.03) 

.00 (.00) 

10.76** 


*p < .05 **p<.01 ns = not significant 

CLASSROOM VISITS 

Analysis of the Classroom Visit data indicates that eMINTS instructional specialists 
spent the largest amount of their Classroom Visit time supporting teachers in 
Lesson Planning and Modeling Instructional strategies. This held true for both years 
of the professional development and both the eMINTS Comp PD and eMINTS4ALL 
programs. Year 1 eMINTS4ALL instructional specialists also spent a large amount of 
time on reflection. Table 11 presents the average percentage of time spent in each 
area by instructional specialists during Classroom Visits, broken down by year and 
program. The first column gives the mean percent time spent in each area, along 
with the standard deviation (SD) in parentheses, in the eMINTS4ALL program in 
Year 1. The second column presents those data for teachers in the eMINTS Comp PD 
program in Year 1, and next two columns represent those programs in Year 2. 


Table 11: Percentage of time spent on different activities during Classroom 
Visits, by program and year 



Year 1 - 4AII 

Year 1 - Comp 

Year 2 - 4AII 

Year 2 - Comp 

Topic Area 

Mean%(SD) 

Mean % (SD) 

Mean%(SD) 

Mean % (SD) 

Modeling 

Instruction 

21 (19) 

17 (12) 

35 (21) 

23 (20) 

Lesson Planning 

30(22) 

39 (19) 

27(17) 

32 (23) 

Technology 

Assistance 

10(13) 

13 (11) 

16 (14) 

15 (13) 

Reflective Practice 

26 (31) 

15 (13) 

12 (10) 

13 (10) 

Problem Solving 

13 (10 

13 (13) 

6(9) 

10(11) 

Other 

1(4) 

4(9) 

5(10) 

8 (10) 


The eMINTS Comp PD program requires more Classroom Visits than the 
eMINTS4ALL program, as evidenced in Table 12. The data indicate eMINTS Comp 
PD Year 1 teachers received an average of two more visits per year than eMINTS 
Comp PD Year 2 participants. 
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Table 12: Average number of Classroom Visits, by program and year 


Group 

Mean Number of Visits to Teachers (SD) 

Year 1 - 4AII 

3.87 (1.9) 

Year 1 - Comp 

9.81 (2.7) 

Year2-4AII 

4.08 (1.1) 

Year 2 - Comp 

7.79 (2.8) 


We also looked to see if the PD4ETS participants and graduates and eMINTS staff 
were doing similar things during their Classroom Visits, and if they made a 
comparable number of visits over the year. We found that the eMINTS staff were 
doing more Lesson Planning than their PD4ETS counterparts, in both eMINTS Comp 
PD and eMINTS4ALL (see Table 13). Meanwhile, PD4ETS participants and graduates 
doing eMINTS4ALL were engaging their teachers in more Reflective Activities than 
staff doing eMINTS4ALL; and PD4ETS participants and graduates doing eMINTS 
Comp PD were Modeling Instruction more than eMINTS staff. In both programs, the 
eMINTS staff conducted more visits to their teachers than PD4ETS participants and 
graduates. 


Table 13: Percentage of time spent on different activities, and number of 
Classroom Visits by program and staff/PD4ETS 



eMINTS4ALL 

eMINTS Comp 


Staff 

PD4ETS 

Staff 

PD4ETS 

Modeling 

Instruction 

25.6% 

28.1% 

16.3% 

25.8% 

Lesson Planning 

38.0% 

22.0% 

39.9% 

26.6% 

Technology 

Assistance 

12.1% 

13.1% 

13.4% 

15.4% 

Reflective Practice 

11.3% 

23.9% 

13.1% 

14.6% 

Problem Solving 

10.3% 

9.2% 

13.4% 

8.1% 

Number of Visits 

5.30 

3.12 

10.10 

6.40 


TEACHER MASTERY 

We also analyzed the Teacher Mastery data generated from our review of the 
artifacts in teacher portfolios, using rubrics for each artifact. The rubrics evaluated 
the teacher work products (Lesson Plans, WebQuests, and Classroom Websites) on a 
1-3 scale, with 1 reflecting low-quality work and 3 high-quality work. Our analysis 
of the Teacher Mastery data indicated that, overall, teachers scored highest on the 
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WebQuest artifact [see Table 14). Their average total scores on this artifact were 
40.74, based on 17 items; this means that teachers scored an average of 2.39 per 
item on their WebQuests, well above the medium quality score of 2. On the Lesson 
Plans, teachers overall scored just about at the medium level. The average total 
score on the Lesson Plans was 41.45, based on 21 items; this means average of 1.97 
per item. Teachers scored lowest on Classroom Websites. The average total score on 
the Classroom Website artifacts was 31.5, based on 17 items; this means an average 
of 1.85 per item. 


Table 14: Average total and per-item scores on teacher portfolio artifacts 


Artifact 

# of items in rubric 

Average total score 

Average per-item score 

Lesson Plan 

21 

41.45 

1.97 

WebQuest 

17 

40.74 

2.39 

Classroom Website 

17 

31.5 

1.85 


One possible reason for the differences in artifact quality is that WebQuests provide 
teachers with a pre-determined structure for planning an online investigation. A 
WebQuest has a distinct format that prescribes what should be included and how 
the lesson should be designed. In addition, there are many online examples of high- 
quality WebQuests that teachers can review. This level of structure may enable 
more teachers to meet the requirements for quality work stipulated in the rubric. 
The eMlNTS program has a constructivist Lesson Plan template that also provides 
teachers with a structure for designing quality lessons, but it is less prescriptive 
than the WebQuest format and therefore results in more variation in quality across 
teachers. Although eMlNTS participants take part in eMINTS PD sessions on how to 
create high-quality Classroom Websites that includes information about what kind 
of content and design elements are important, teachers have a great deal of 
flexibility in the websites they design, so there is a great deal of variation in their 
quality. 

More interesting, perhaps, is an examination of the item-by-item frequencies for the 
three artifacts, which provide a greater level of detail about what concepts teachers 
have mastered and which they have not at the end of their eMlNTS experience. The 
Lesson Plan scores suggest that teachers were skillful in articulating what students 
need to do to demonstrate their learning, connecting their lesson to the standards, 
requiring students to create an authentic end product, identifying learning goals, 
and using technology effectively. Where they struggled the most was again in 
designing the lesson to accommodate student diversity (in background, language, 
ability and learning style), articulating how students will share resources, having 
students generate questions to guide their learning, helping them plan their 
explorations, and helping them reflect on their learning. 
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The WebQuest scores show that teachers were most successful at sharing their 
assessment resources with the students and having clear assessment criteria that 
were connected to the lesson, providing a clear and coherent task and process for 
completing the task, and requiring students to be creative in the way they 
communicated their answer to the WebQuest lesson. The only items on which 
teachers performed poorly related to the use of deliberate grouping strategies 
(assigning students different roles in group projects), designing lessons to 
accommodate student diversity [providing students with options for different 
resources, processes, and end products), and concluding the lesson in a way that 
connects what they have learned to other lessons. 

The Classroom Website scores indicate that teachers' websites were generally well 
designed and easy to navigate, and provided links to many high-quality resources. 
However, they often did not contain student work, information about or examples of 
assessments used in their teaching, or information about classroom policies. 

RELATIONSHIP BETWEEN TEACHER MASTERY SCORES AND 
TEACHER FOLLOW-UP INTERVIEWS 

As noted above in the Methods section, EDC evaluators conducted interviews with 
sixteen randomly selected participants in the spring after they completed their 
eMlNTS professional development programs. Ten of the interviewees had 
completed the eMlNTS Comprehensive professional development, and six had 
completed the eMlNTS4ALL professional development. To focus our discussions 
with teachers on concrete ideas and instructional practices, we based our interviews 
on artifacts. The artifacts we chose as the anchor for our interviews were the Lesson 
Plans teachers submitted as part of their eMlNTS portfolios. Though teachers were 
welcome to refer to other lessons or general practices in the interviews, the 
portfolio Lesson Plans gave us a common object around which to ground our 
discussion. Here we discuss the relationships between the findings from the Teacher 
Mastery analysis of the Lesson Plans and our analysis of the interview data 
regarding some of the key topics addressed in the eMlNTS program. In many cases 
the interview data corroborate the Teacher Mastery findings, but in other cases 
interview data suggest that the larger picture of what teachers take away from 
eMlNTS is more complex than the one presented by Teacher Mastery scores alone. 

TECHNOLOGY USE 

Helping participants understand how to incorporate technology into instruction to 
support student learning is one of the cornerstones of eMlNTS. The Teacher Mastery 
scores teachers received on the Technology Use item of the Lesson Plan rubric 
demonstrate that a majority of teachers (64.1%) designed Lesson Plans that 
integrated technology to support students' learning activities, rather than just to 
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increase productivity or to give students an opportunity to practice technology 
skills. 

The follow-up interviews provide additional evidence that teachers gained from the 
professional development an understanding that technology should serve as a 
support for learning and its use should not be an end in itself "I learned it's not 
teaching for the technology, but with the technology," said one teacher. "I used to 
teach for it, but eMINTS helped me just integrate it. eMINTS was very hands-on and 
allowed us to make things we could use with children." Another teacher added, "The 
great part about eMINTS is that it’s not just technology, it’s a framework for 
teaching." 

Fourteen of the sixteen teachers interviewed gave specific examples of how they 
were using technology to help students engage with content in new ways. The 
following is an example of how one teacher integrated technology into a math lesson 
to help students understand a complex concept. 

The kids last year struggled a lot in general, so to have something where they 
used the Internet, that motivated them. This was a lesson out of their textbook, 
which I tweaked. They could play around with the equations and lines much 
quicker [using the software], and see what happens when they changed things. 

With my honors group, they tried to figure out how to make an undefined line. 

They were creating their own shapes. 

Interestingly, eMINTS4ALL participants were just as likely as eMINTS 
Comprehensive participants to talk about how the professional development not 
only helped them think about using technology in their teaching, but also allowed 
them to engage in different instructional practices. One eMINTS4ALL teacher 
observed: 

I realized how powerful having an interactive SMART Board™ in the class was. 

It's easy to work and use . . . I've really found the technology to be helpful in my 
classroom. Before eMINTS I didn't use as many virtual manipulatives on 
websites. We just didn't do that stuff because we didn't have good computer 
access in labs. Now my students are making PowerPoints on ecology, and they 
are doing Internet research while working in the library. They can then be very 
self-guided. They couldn't really do this before without the technology. 

As enthusiastic as the participants were about the technology they received through 
the program and learned how to use, none of the teachers interviewed spoke about 
technology as the answer to all of their instructional needs. Instead, participants 
came away from the program with positive but realistic expectations of what 
technology could contribute to learning. As one teacher said: 
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I learned a lot about technology . . . You'll never replace a good book ... or 
pencil and paper math, the old fashioned techniques, but the eMINTS program 
gives us opportunities to explore things in more than one way— and that's what 
education is all about. 

COLLABORATIVE GROUPING 

The eMINTS program addresses collaborative learning not only by covering it in the 
professional development session content but also by having instructional 
specialists model the use of this technique in their own instruction. The Teacher 
Mastery scores from the Lesson Plan rubric indicate that, while most teachers 
(81.3%) had students work in groups on their lessons, only 11.5% of teachers had 
students assume different but interdependent responsibilities in their group work. 

The majority of interviewees reported being experienced with group work. Eleven 
out of the sixteen interviewees said that they used group work often in their 
instruction even before eMINTS, but nine of the sixteen noted that eMINTS had 
introduced new techniques for grouping that they then applied in their teaching. 
Some teachers reported having students work in small groups more often since the 
professional development. One teacher stated: 

I do a lot more collaborative work and divide them up differently than before 
[eMINTS]. For example, I now have students read pieces of a newspaper, then 
get back together to discuss it. They would each present an article. I hadn't 
done this type of thing before. The kids are more interested listening to each 
other, and benefit from putting things in their own words. 

Other teachers mentioned that eMINTS gave them more ideas about how to 
formulate groups. One teacher said. "Actually this year I bought [a commercial 
product for team work], and it creates groups for you based on ability, or random, or 
other criteria. That was something I saw at an eMINTS conference. Before I grouped 
completely randomly; now I'm more picky.” 

Another teacher described how eMINTS helped her improve her grouping 
techniques. 

So, the training helped me with some strategies to make sure the groups were 
actually functioning. Like making sure that each person was being responsible— I 
had created some ways of making sure that they did that but eMINTS did give 
me a few more ideas on that. 

One teacher observed that some of the grouping strategies suggested by eMINTS 
helped her support different kinds of learners. 

From eMINTS I learned to allow more time for discussion. Some thinkers are 
great at talking about their ideas but not [at] writing it down. There is writing in 
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math, and they need to sit and chat with each other; that "think, pair, share" 
strategy is really important. They need to think through what they are doing and 
talk about it . . . That is a strategy I learned from eMINTS that I do all the time. 

They love to talk things out first, and then they can get it on paper. If you look at 
the lesson, there is a lot for them to look at, and they could feel overwhelmed, 
but by giving them discussion time, it's not so overwhelming. 

Still another teacher reported that eMINTS changed the way she thought about 
resolving conflicts that arise in group work. 

Something I learned from eMINTS, there are always kids who don't work well 
with others. Before I would always blame the student. eMINTS taught me, you 
need to say to the group, what are you doing to make that student feel part of 
the group? That really helped a lot. I focus on the group and not the child that's 
causing problems. 

While most teachers were familiar with grouping students, only three of the sixteen 
teachers interviewed mentioned having the students assume roles in their group 
work. One teacher stated: 

I learned about having the students take different roles in group work from the 
eMINTS instructional specialist, during a module where we looked at different 
WebQuests. It made a big difference for them to have a specific job. It seems 
more realistic. I didn't do that before. 

Another teacher who already knew the importance of roles believed that eMINTS 
reinforced her understanding. 

I was basically very much trained in constructivism, so a lot of the lessons to me 
were refreshers. It reaffirmed the idea that they had to have specific roles and 
choosing group sizes that are meaningful to whatever project you're doing. You 
want everybody to have a meaningful role, not just busy work. The eMINTS 
program does a real good job teaching people how to do that and how to 
manage groups and use strategies that work. 

Even among those who spoke about students having different roles, only one 
teacher mentioned the need to create interdependence among students working in 
groups. 

There are a lot of strategies for making group work happen. From the beginning 
[of eMINTS] we talk about behavior within a cooperative group, whether it is a 
partner or a group of four people. We talk about roles and how they can change 
daily, how to share use of the computer in an equitable way, how to solve 
disagreements when they arise. We use an "ask three then me" strategy for 
making sure students become interdependent on one another rather than just 
relying on me for answers. We try to make the roles very definitive— who is 
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responsible for what piece of the puzzle. And the key for me is that in some 
assignments the grade is given as a cooperative grade, but in others it is an 
individual grade. I don't want to rely too heavily on the group process if I'm 
going to grade a project. 

The interview analysis suggests that even those teachers who are comfortable and 
familiar with grouping students took away new ideas about grouping strategies 
from eMlNTS. However, the Teacher Mastery data, coupled with the interview data, 
also suggest that teachers are not necessarily coming away from the professional 
development with a clear understanding of the importance of creating 
interdependence among group members by assigning different, meaningful roles 
and responsibilities to students. 

ACCOMMODATING STUDENT DIVERSITY 

The eMlNTS program goals include not only helping participants to address student 
diversity in their instruction but also helping them to understand that students can 
be diverse in many ways, such as their learning styles, their abilities, their 
languages, and their races, ethnicities, and backgrounds. The Teacher Mastery 
scores on the Lesson Plan rubrics indicate that the majority of participants (63%) 
addressed diversity at least in a limited way (receiving a score of 2) in their Lesson 
Plans, but it is notable that very few (4.2%) took into account multiple forms of 
diversity, such as learning styles, ability, and multicultural diversity. Furthermore, 
very few (1%) accommodated diverse learners in their Lesson Plans by offering 
different activities or different products through which students could demonstrate 
their learning. 

In the follow-up interviews, we asked teachers what they had learned from eMlNTS 
about addressing diversity. Six of the teachers mentioned that eMlNTS had helped 
to expand their conception of diversity and given them strategies for addressing 
diversity in their classrooms. "A big part of eMlNTS for me was widening the 
definition of diversity,” one participant stated. Another teacher noted, "I had thought 
about diversity in limited ways, but eMlNTS made me think about it and apply it in 
new ways." 

One notable pattern in the data was that, in talking about how they addressed 
diversity, ten out of the sixteen teachers interviewed spoke about how they grouped 
students of different abilities or learning styles. This was the most commonly cited 
strategy for addressing diversity among the interviewees. Teachers were able to 
point to specific lessons they learned. One teacher gave an example of a technique 
she learned from eMlNTS about grouping by ability level. 

I . . . learned from eMlNTS that when you group a high student with a low 
student they can get very impatient. Now I try to pair an average student with a 
low student. They are more patient. You have to have different abilities in a 
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group, but it's often better to have your high paired with an average student, 
and low with an average child. 

Half of those interviewed discussed how technology could help accommodate 
diverse abilities and needs. One teacher stated: 

[I learned] just to be aware that when you choose technology [you need] to take 
learner diversity into account. When you pick a website, it needs to be a reading 
level that all students are comfortable with. My ESL student could do the 
[software I chose for the lesson] because it was visual. eMINTS just reminds you 
to make it equitable. 

Another teacher described how technology can help support students with different 
strengths and weaknesses. 

I think [technology] builds confidence for lower-education kids. I thought 
technology would make it more difficult for them, but that was an "ah ha" 
moment. It's given them a great deal of confidence. It's amazing how students 
who are struggling in school can figure out a computer game. That's where the 
technology comes in. A lot of what special education kids struggle with is 
organization, and the technology helps with organization. 

Only one of the interviewees said that she addressed diversity by offering different 
activities for different students to engage in or providing students with options for 
creating different work products to demonstrate their learning. Three teachers said 
that diversity was not an issue for them because their schools lacked ethnic 
diversity. "1 remember [the eMINTS instructional specialists] talking about 
[diversity],” said one of these teachers, "but I work in a county that’s all white, with 
no diversity. There is no practical application in my classroom." Comments such as 
this suggest that a few teachers still held a limited view of what is meant by 
diversity. 

INQUIRY-BASED LEARNING 

Inquiry-based learning is strongly emphasized in the eMINTS program; a number of 
its professional development sessions are designed to help teachers learn how to 
teach using an inquiry-based approach. There are many instructional components to 
inquiry-based learning, including having students develop their own questions and 
engage in independent research, and asking students to analyze and synthesize the 
data they collect and reflect on their learning. Using the inquiry-based instructional 
approach effectively requires teachers to master a range of skills and techniques, a 
task that can prove challenging for many teachers. 

The Lesson Plan rubric has six items that address aspects of inquiry-based learning. 
Teacher Mastery scores reflect a high level of integration of some of these concepts 
into teachers' Lesson Plans, but a low level of integration of other concepts. For 
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example, slightly more than half (51%) of the teachers who submitted Lesson Plans 
came up with high-quality essential questions that spur inquiry, require students to 
gather and analyze data and involve an authentic problem. More than a third 
(39.1%) described specific techniques students could use to organize the 
information they gathered and less than a third (31.3%) described the techniques 
students could use to analyze the information they collected. However, a large 
majority (74%) did not require students to generate their own questions and 68.2% 
had no activities in the lesson that involved student reflection. 

The interview data show that, while many teachers recognized the benefits of the 
inquiry-based learning approach, they also saw the challenges of it, and some 
struggled to implement it. Six of the teachers interviewed had not used inquiry- 
based learning methods before in their teaching, and seven of the sixteen spoke of 
the difficulties of implementing these methods. However, most saw the value of this 
method, because eleven of the sixteen teachers interviewed reported using inquiry- 
based teaching methods more often in their teaching since eMINTS. "I do inquiry- 
based learning informally all the time," said one teacher. "That came from the 
eMINTS training. I am very animated when 1 teach. 1 come up with stories. 1 did that 
in student teaching, but it was me force-feeding and kids push back. They don't push 
back as much when you let them think first." 

Another teacher described how eMINTS expanded on the ideas about inquiry-based 
learning she encountered in her pre-service teacher education. 

Even with kindergarten, this kind of teaching [inquiry-based learning] provides 
ownership of work by [giving students] some control about what they are 
learning, and the students are more likely to want to be a part of it. I learned 
some about this in college but eMINTS emphasized this; and it gave me a finer 
look at things. College was much more macro and wide. eMINTS focused it in to 
my classroom and learners and refined my ideas. 

A third teacher mentioned how eMINTS extended what she was she was already 
doing in her teaching by emphasizing reflection and questioning. 

That reflection on their own work is pretty key. They need to go back and reflect 
and see what they did right and what they did wrong, and how to improve it. 

They won't grow if they don't do that. I didn't do enough of that before I started 
eMINTS. I did a lot of discovery-based learning, which is similar to eMINTS 
inquiry-based learning, but not exactly the same thing. So, where they were 
having to discover their learning and discover the answers— they didn't 
necessarily have to make up their own questions. And in 6**^ grade they're 
getting really the first taste of doing research. We give them a really big project 
and I give them a plan. And they come up with an additional question to add to 
the questions I've already asked. It's kind of a start for them into the research 
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project and the research process. And I think I would like to have them start 
coming up with more of their own questions but I'm still going to give them the 
guide that I give them as well. Because they won't answer the questions that I'm 
looking for on their own. They need a little scaffolding on that part. 

As noted earlier, Teacher Mastery scores for designing lessons that involved 
student-generated questions were generally low. One of the main obstacles to the 
use of inquiry-based methods that teachers cited was the difficulty of getting their 
students to ask good questions. While the teacher above described how she’s just 
starting to introduce student-generated questions into her teaching, other teachers 
admitted that they struggled with this. One teacher stated: 

Well, eMINTS was my first introduction to inquiry-based learning. I have not 
mastered the skill of it. I have not been able to get those deeper questions from 
my students. The self-generated questions that my kids generate are just kind of 
knowledge- and content-based. They're not higher-level thinking questions. I 
feel that it's one of the weakest links in myself that I don't have the skill to bring 
it out in them. 

Another teacher felt that it was especially difficult to elicit questions from lower- 
achieving students. 

Inquiry-based learning, that's something I struggle with— especially with larger, 
more diverse classes. Lower students have a hard time forming their own 
questions. I know I need to work on how to help these students do these things. 

But eMINTS helped give me some ideas. For example, we are doing a 
PowerPoint project on ecology. I gave them some basic ideas, but they needed 
to come up with the content and points to discuss on the PowerPoint through 
research. The high-level ones do well; the rest struggle from lack of practice. 

They struggle with forming questions. 

However, even teachers who noted that some students have difficulty generating 
questions still believed the process was valuable. 

The students generated their own ideas for inquiry and researched and 
gathered data. Some responded better than others though. Some are lazier and 
not motivated by it and moan and groan and just ask for problems to do 
instead. This takes a lot more work and motivation, but down the line most saw 
the benefit of it by understanding the material better and seeing a reason for 
learning it. In math, sometimes they don't see the application of, for example, 
quadratic equations, but this helps. Plus this makes it more than just out of a 
textbook, which they like. They get to explore and learn. 

We found it particularly interesting that participants seemed to closely associate 
eMINTS with inquiry-based learning. Whenever they talked about the eMINTS 
philosophy, or used "eMINTS” as an adjective describing a way of teaching, the 
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meaning was usually synonymous with inquiry-based learning or some aspect of it. 
For example, one teacher stated, "Mostly my content lessons are taught in the 
eMlNTS fashion, with inquiry-based methods.” Another teacher said, "Since eMINTS 
I've added [inquiry-based learning] a lot more. Even before I got my computers after 
eMINTS, I was preparing my Lesson Plans to match eMINTS, so once they arrived, I 
changed ever}Ahing to match the eMINTS style.” 

In responding to a question about whether she would revise a lesson she had used 
with her students, another teacher used the term "eMINTS” to mean an inquiry- 
based approach to teaching. 

I think on the whole it's a good lesson for accomplishing what I want them to 
accomplish. I would probably give students more opportunity to participate. 

Maybe I would let students be the camera people. I also probably spent a lot of 
time downloading the pictures to the computer. I could give that task over to 
the students. I would give students more opportunity to do things for 
themselves. In 5th grade, I'm probably the first teacher to let them do that. That 
would be one of the "eMINTS" things I would do. 

The interview data indicated that teachers identified eMINTS with inquiry-based 
instruction. While many expressed enthusiasm for this approach to teaching, there 
was also an acknowledgment that this approach could be challenging, and some 
teachers questioned whether young students or low-achieving students were 
capable of handling the responsibility of having control over their own learning. 
However, teachers who had integrated these strategies saw the positive impact of 
this approach, and appreciated eMINTS for providing them with practical strategies 
for implementing this kind of teaching in their classroom. 

ASSESSMENT 

The eMINTS program encourages participants to think about assessment, not only 
as a final test or culminating product at the end of a lesson, but also as a variety of 
activities that can occur throughout a lesson to help both the teacher and the 
students understand what students are learning. The professional development 
sessions also emphasize the need for teachers to ask their students to analyze and 
synthesize the information they gather through inquiry and to present it in 
authentic and creative ways. The Teacher Mastery scores on the Lesson Plan rubric 
indicated that teachers were able to integrate some of these ideas into their lessons, 
but struggled with others. Over half [58.3%) of teachers had Lesson Plans that 
required students to create "an authentic end product” that could communicate to 
"multiple audiences.” A third (32.8%) of the participants had final assessments or 
end products that required students to engage in analysis and synthesis. However, 
less than a quarter (22.9%) shared or co-created their assessment criteria with 
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students, and almost half (49%) identified only one or no final assessments / 
products in their Lesson Plans. 

Though Teacher Mastery of the eMINTS approach to assessment is mixed on the 
Lesson Plan rubric, the interview data paint a somewhat different picture. The 
majority of interviewees reported using multiple assessments. Fourteen of the 
sixteen teachers interviewed described using a wide a variety of assessments with 
their students for lessons. One teacher listed all the kinds of assessment she 
included in her lesson, "1 had a scoring rubric. Students composed similes, drew 
pictures, did self-evaluation, and held consistent writing conferences with me.” Nine 
of the sixteen specifically noted that eMINTS changed their thinking about 
assessment. An elementary teacher described how eMINTS influenced her approach 
to assessment. 

The students had a performance task, in the experiment, and they had 
worksheets and "think and write" questions and when they [used the software 
in the lesson], they had to justify orally why they picked the characteristics. We 
also had a debriefing time at the end and they journaled. They made a postcard 
out of the animals and put it on our blog, so parents could see. Before eMINTS I 
did paper-and-pencil multiple-choice tests. With eMINTS, I use performance 
tasks and use a rubric. A performance task is a much better way to assess 
students. 

A math teacher also spoke about how eMINTS changed her approach to assessment. 

I've tried to incorporate multiple assessments. I used these in all my Lesson 
Plans once I got into the eMINTS swing, to not just have a test at the end, but 
this is very different from what most math classes are like. I use rubrics that 
show students all the points they can get from the ongoing activities that make 
up one lesson or unit. This is really something new I learned in eMINTS— 
multiple assessments. Most math is just a quiz in the middle and test at the end. 

My kids were used to that and this was very different for them. 

Another teacher detailed how the different kinds of assessments she used served 
different purposes, and how her students responded to this approach to assessment. 

I gave the students a scoring guide up front. There was a written exam, but we 
did ongoing assessments on the SMART Board™ after each session. That is 
different [since eMINTS]. For example, after the food pyramid lesson, we'd do 
10 questions multiple choice; students would work as a group and discuss them. 

We also did individual assessments to check their progress on understanding. 

The ongoing assessments were new, mostly we had done product assessment, 
but being able to do ongoing check-ins was great. My students enjoyed using 
the SMART Board™ and talking over their answers. They also liked having more 
than one grade, not just an end assessment. 
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Other teachers echoed this teacher’s description of the positive reaction students 
had to multiple assessments. Here is one example. 

I used assessments throughout this project. I had them present their problem, 
getting things done on time. They had a rubric with lots of items they could get 
points for, an oral presentation. A lot of my students liked doing the projects 
because they could do better grade-wise because there were more 
opportunities to score points, rather than one test. They were rewarded for 
effort. They responded well to that. 

Another teacher believed the multiple forms of assessment enabled a wider range of 
students to feel successful. 

I started to pick up a lot of the formative and informal assessments. What I like 
about it is that your hard-working students can be rewarded with informal 
assessments, as opposed to just smart students. I didn't really consider this 
before eMINTS. 

Teachers also talked about using simple assessments flexibly to monitor student 
understanding on a regular basis. "We do shorter, common assessments just to see if 
students are getting it before moving on," said one teacher. "They have some 
practice online tests that are like games." Another teacher noted, "Sometimes I just 
give them a checklist for working and staying on task. I give them guides ahead of 
time to show them what we are looking for, having it ongoing — 'how did you work 
each day?' — then evaluate how to improve." 

Some teachers noted that eMINTS helped them understand how to create and use 
rubrics as part of their assessment strategy. One participant described collaborating 
on rubric design with her students. 

I learned about creating a rubric with students from eMINTS. They said to have 
the students help you create the lesson, so I did the same with the rubric. I used 
to create my own rubrics, but not with the students. My lessons and rubrics 
used to be about me, because I created them, but now they are created with 
the students. They tend to follow the rubric because they created it. They would 
talk to each other about the rubric as they worked on their projects, making 
sure they covered everything. 

Another teacher mentioned that eMINTS gave her an idea of how to use a rubric to 
integrate peer assessment of group process into her overall assessment strategy. 

One of the things I learned from eMINTS was to give them a rubric for 
discussion of group work, which I never thought of before eMINTS. Did my 
partner listen to me? There are all those factors, that list of things that make for 
good group work, and they assess each other on that. If they are doing group 
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work and know they are getting a grade, they are annoyed if others aren't 
working. So I have them assess each other on their group work. 

It is unclear why teachers were so articulate and descriptive about their use of a 
wide range of assessments in interviews, but did not score high on the assessment 
items of the Lesson Plan rubric. Perhaps they simply did not include all of the 
assessments they used in their teaching in their written plans. It is noteworthy that, 
of all the different topics covered in the professional development sessions, 
interviewees reported that eMINTS had the greatest impact on their Use of 
Technology and their approach to Assessment. Perhaps because strategies such as 
ongoing assessment were new to teachers, they were not accustomed to including 
them in formal Lesson Plans, even if they did use them in their teaching. The fact 
that interviewees were so detailed in their descriptions of the assessments they 
used in their practice suggests that teachers may have mastered this concept more 
than their Teacher Mastery scores reflect. 


RELATIONSHIP BETWEEN PD FIDELITY AND TEACHER MASTERY 

To understand whether there was a relationship between the fidelity of the 
professional development experienced by teachers and the level of mastery they 
displayed in their portfolio artifacts, we ran correlations examining this 
relationship. We found significant correlations between both composite fidelity 
scores and the quality of the Lesson Plan artifacts. The correlation between the 
WebQuest and the Program Construct Fidelity composite was marginally significant 
and the relationship with the Factor Analysis-Based Fidelity [FA-Based Fidelity) also 
approached significance (see Table 15). Both types of fidelity have a significant 
correlation with the Lesson Plan scores, which is important since this is the artifact 
that is probably most representative of teachers' instructional practice. These 
results provide evidence that the quality of the professional development does have 
an impact on the kind of work that teachers produce. 


Table 15: Correlations between Teacher Mastery and PD Fidelity 
Composites 



Lesson Plan 

WebQuest 

Factor Analysis-Based Fidelity 

.256** 

.196* 

Program Construct Fidelity 

.284** 

.190** 


**p < .01; 'p = .05; "p < .06 


We then ran correlations between Teacher Mastery and the different factors that 
comprise the two Fidelity composites (see Table 16). This analysis indicated that the 
professional development factors most strongly associated with Teacher Mastery 
include, among the factor analysis-based factors: Scaffolding Instruction, Facilitating 
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Discussion and Active Work/Learning; and among the Program Construct Factors: 
Modeling Instruction, Technology Utilization, Connection to Practice, and Inquiry- 
based Learning. This analysis also shows that one factor per composite was also 
significantly correlated to Teacher Mastery on the WebQuest — Facilitating 
Discussion in the FA-Based Composite and Modeling Instruction on the Program 
Construct Composite. 

Table 16: Correlations between Teacher Mastery and PD Fidelity Factors 


Composite 

Factor 

Lesson Plan 

WebQuest 

FA-Based Fidelity 

Scaffolding Instruction 

.263** 

ns 


Facilitating Discussion 

.174* 

.238* 


Active Work/Learning 

.296** 

ns 

Program Construct 
Fidelity 

Modeling Instruction 

.388** 

.282** 


Technology Utilization 

.268** 

ns 


Connection to Practice 

.217** 

ns 


Inquiry-Based Learning 

.205** 

ns 


*p < .05 **p < .01 ns - not significant 


We then ran statistical regressions on these same factors, as a more rigorous test of 
which factors are predictive of Teacher Mastery on the different artifacts. We found 
that some of the factors above were no longer significant. However, the regression 
demonstrated that, among the Factor Analysis-Based Fidelity components. Active 
Work/Learning continued to be a significant predictor of Teacher Mastery on the 
Lesson Plans [Betai° = 0.26, t = 2.5, p < .02). This finding indicates that one of the 
key components of the eMINTS model — having teachers work together to take an 
active, hands-on approach to what they are learning — is one of the most important 
factors in shaping program impact on teachers. Another regression run with the 
Program Construct Fidelity factors demonstrated that Modeling Instruction was a 
significant predictor of Teacher Mastery on the Lesson Plan and the WebQuest [Beta 
= 0.43, t = 3.3, p < .001, and Beta = 0.57, t = 3.2, p < .01, respectively). 

RELATIONSHIP BETWEEN PD FIDELITY AND THE TEACHER 
TECHNOLOGY LITERACY SURVEY 

We treated the technology survey responses as dependent variables that could be 
affected by the quality of the eMINTS professional development. When we ran 
correlations between the two datasets, we found that there were significant 

Beta is a measure of how strongly one variable predicts another in a linear regression, derived from the 
slope of the relationship. 
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negative correlations between both fidelity composites and survey respondents' 
identification of having computer issues and support issues (see Table 17). This 
means that teachers who experience high-fidelity professional development report 
experiencing fewer computer and support issues. 


Table 17: Correlations between PD Fidelity and Teacher Survey 



FA-Based Fidelity 

Program Construct Fidelity 

Computer Issues 

-.335** 

-.344** 

Support Issues 

-.200* 

- 210** 


*p < .05 **p < .01 


RELATIONSHIP BETWEEN TEACHER MASTERY AND CLASSROOM 
VISITS 

We then looked at the relationship between Teacher Mastery of eMINTS concepts 
and Classroom Visits — what activities took place during the visits and how often 
those visits occurred. We examined the data for each program [eMINTS 
Comprehensive and 4ALL) separately because the expectations for the number of 
visits are different for the two. We found (see Table 18) that the eMINTS4ALL 
participants showed a significant positive correlation between the time spent on 
Lesson Planning and teachers’ scores on their Lesson Plans, and a significant 
negative correlation between the amount of time spent on Technology Assistance 
and the quality of teachers’ Lesson Plans. This means that teachers who spent more 
time with their instructional specialists working on Lesson Planning ended their 
professional development with higher-quality Lesson Plans, while those who used 
their Classroom Visit time to have instructional specialists provide Technology 
Assistance produced lower-quality Lesson Plans. We found no significant 
correlations between Classroom Visits and Teacher Mastery for the eMINTS 
Comprehensive participants. 

Table 18: Correlations between Classroom Visits and Teacher Mastery for 
eMINTS4ALL participants 



Lesson Plan 

Lesson Planning 

.364** 

Technology Assistance 

-.326** 


**p < .01 
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RELATIONSHIP BETWEEN CLASSROOM VISITS AND FIDELITY 


We next examined the relationship between the Classroom Visit activities for each 
program [eMINTS Comprehensive and 4ALL) and the components of Program 
Construct Fidelity and Factor Analysis-Based Fidelity. The analysis identified a 
number of significant relationships (see Tables 19-22]. Generally, these tables show 
negative correlations between Modeling Instruction and overall fidelity across both 
programs. This indicates that the stronger the fidelity of the professional 
development, the less time instructional specialists need to spend on Modeling 
Instruction during their Classroom Visits. For the eMINTS4ALL participants, there 
were strong positive correlations between Lesson Planning and fidelity, indicating 
that teachers who experienced higher fidelity PD could focus on improving their 
Lesson Plans during Classroom Visits. The eMINTS4ALL participants also had strong 
negative correlations between Technology Assistance and fidelity, indicating that 
teachers who experienced better fidelity could spend less time dealing with 
technical issues. The eMINTS4ALL participants also had strong positive correlations 
between Reflective Practice and fidelity, indicating that teachers who experienced 
better fidelity could reflect more on what was happening in the classroom. This 
pattern also held for Problem Solving. For all groups, the correlations between 
fidelity constructs and number of visits were strongly negative. It is unclear why this 
is the case. It is not likely that there is a causal connection between the two. This 
may be a finding in need of further investigation. 


Table 19: Correlations between Classroom Visits and Program Construct 
Fidelity for eMINTS4ALL participants 



Overall 

Fidelity 

Program Construct Fidelity 

Classroom Visit 
Activities 


Modeling 

Instruction 

Community 

Building 

Technology 

Utilization 

Connection to 
Practice 

Inquiry-Based 

Learning 

Modeling 

Instruction 


-.60*** 

-.25* 

-.36** 

59*** 

59* * * 

Lesson Planning 

35** 

31** 

ns 

ns 

35** 

00 *** 

Technology 

Assistance 

-.34** 

ns 

-.27* 

33** 

31** 

31** 

Reflective 

Practice 

.66*** 

* * 

* * 

^ 2 *** 

* * 

00 *** 

Problem Solving 

* 

ns 

.26* 


42 * * * 

.31** 

Number of 
Visits 


-.25* 

55* * * 

54* * * 

02*** 

32** 


*p < .05 **p < .01 ***p < .001 ns = not significant 
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Table 20: Correlations between Classroom Visits and Program Construct 
Fidelity for eMINTS Comprehensive participants 



Overall 

Fidelity 

Program Construct Fidelity 

Classroom 
Visit Activities 


Modeling 

Instruction 

Community 

Building 

Technology 

Utilization 

Connection 
to Practice 

Inquiry-Based 

Learning 

Modeling 

Instruction 

-.17* 

ns 

-.20** 

ns 

43*** 

38* * * 

Lesson 

Planning 

ns 

21** 

ns 

21** 

-.15* 

ns 

Technology 

Assistance 

ns 

2g* * * 

ns 

ns 

ns 

ns 

Reflective 

Practice 

ns 

ns 

ns 

ns 

ns 

ns 

Problem 

Solving 

ns 

ns 

.17* 

32*** 

ns 

ns 

Number of 
Visits 

_ 42*** 

19** 

28* * * 

30* * * 

ns 

ns 


*p < .05 **p < . 01 ***p < .001 ns - not significant 


Table 21: Correlations between Classroom Visits and the Factor Analysis- 
Based Fidelity for eMINTS4ALL participants 



Overall 

Fidelity 

Factor Analysis-Based Fidelity 

Classroom Visit 
Activities 


Structured 

Activities 

Participant-Led 

Discussion 

Scaffolding 

Instruction 

Facilitating 

Discussion 

Active 

Work/Learning 

Modeling 

Instruction 

33*** 

63*** 

ns 

59* * * 

-.27* 

42*** 

Lesson 

Planning 

.34** 

32*** 

ns 

.25* 

ns 

42*** 

Technology 

Assistance 

33** 

ns 

-.36** 

-.23* 

ns 

33*** 

Reflective 

Practice 

.65*** 

38* * * 

40* * * 

.66*** 

40** 

47*** 

Problem 

Solving 

40*** 

ns 

40* * * 

.31** 

ns 

48* * * 

Number of 
Visits 

32*** 

37* * * 

42*** 

-.54** 

47*** 

-.30** 


*p < .05 **p < .01 ***p < .001 ns - not significant 
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Table 22: Correlations between Classroom Visits and the Factor Analysis- 
Based Fidelity for eMINTS Comprehensive participants 



Overall 

Fidelity 

Factor Analysis-Based Fidelity 

Classroom Visit 
Activities 


Structured 

Activities 

Participant-Led 

Discussion 

Scaffolding 

Instruction 

Facilitating 

Discussion 

Active 

Work/Learning 

Modeling 

Instruction 

-.16* 

24* * * 

-.17* 

- 20** 

19** 

ns 

Lesson 

Planning 

ns 

ns 

ns 

ns 

ns 

- 20** 

Technology 

Assistance 

ns 

ns 

ns 

ns 

ns 

.15* 

Reflective 

Practice 

ns 


ns 

ns 

ns 

-.16* 

Problem 

Solving 

ns 

ns 

22** 

ns 

-.16* 

ns 

Number of 
Visits 

42*** 

ns 

32*** 

29* * * 

34* * * 

26* * * 


*p < .05 **p < .01 ***p < .001 ns - not significant 


RELATIONSHIP BETWEEN PROGRAM AND TEACHER FINDINGS 
AND STUDENT MAP SCORES 

The next step in the analysis was to examine the relationship between these same 
variables and student outcomes on the Missouri Assessment Program (MAP) test. 
Overall analyses were run to determine any relationships between program factors 
and students’ MAP scores in Communication Arts (CA), Mathematics (MA), Science 
(SC), and Social Studies (SS) across grades 3 through 8, where students had MAP 
data and comparisons were appropriate. Analyses were run overall and controlling 
for the relevant covariates were appropriate [see next subheading). Where sample 
size would allow, analyses were also run independently within districts. 

OVERVIEW OF COVARIATES 

Students' academic scores are often related to demographic variables such as free 
and reduced lunch status. If such variables affect student scores, they may need to 
be statistically controlled when performing analysis on student-based data. Thus 
our first analysis aimed to discover any such covariates — variables that show 
differences in student scores across groups. Student variables that were analyzed to 
check for differences in MAP scores included lEP/LEP status free and reduced lunch 
status (ERL), grade, gender, race, and students’ 2006 MAP scores. 
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lEP/LEP 

lEP (individualized learning program) or LEP [limited English proficiency) status 
may have an impact on the overall mean of MAP scores if students with these 
designations score significantly lower than other students. For this reason, we ran t- 
tests to compare lEP and LEP students to the general population to see if this 
demographic would need to be added as a control variable. As seen in Table 23 and 
Table 24, lEP and LEP students scored significantly lower on every MAP test 
compared to students in the general education program, with t-tests ranging from 
5.5 [df = 3,812) to 42.8 [df = 12,892), with every t-test being significant at p < .001. 
To ensure that this effect was not contingent upon grade level, the same analysis 
was run on each grade individually. lEP students and LEP students scored 
significantly lower on every MAP test in every grade for which there were sufficient 
data to run an analysis. Thus the effect was universal and did not vary across grade. 


Table 23: Comparison of MAP scores for lEP students and students in the 
general education program 


Test 

lEP 

N 

Mean 

SD 

t-value 

P 

Communication 
Arts (CA) 

Yes 

11255 

675.5 

35.1 

42.8 

<.001 

No 

1639 

634.7 

42.4 

Mathematics 

(MA) 

Yes 

11313 

674.8 

44.7 

34.5 

<.001 

No 

1656 

634.2 

45.2 

Science (SC) 

Yes 

3307 

684.7 

45.5 

15.0 

<.001 

No 

507 

652.1 

45.2 

Social Studies (SS) 

Yes 

3446 

690.1 

32.8 

16.6 

<.001 

No 

461 

662.8 

35.5 
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Table 24: Comparison of MAP scores for LEP students and students in the 
general education program 


Test 

LEP 

N 

Mean 

SD 

t-value 

P 

Communication 
Arts (CA) 

Yes 

12616 

671.1 

38.2 

16.2 

<.001 

No 

278 

633.5 

37.4 

Mathematics MA) 

Yes 

12677 

670.5 

46.5 

14.3 

<.001 

No 

292 

631.3 

43.1 

Science (SC) 

Yes 

3782 

680.7 

46.6 

5.5 

<.001 

No 

32 

635.4 

45.8 

Social Studies (SS) 

Yes 

3869 

687.2 

34.1 

6.8 

<.001 

No 

38 

649.3 

36.0 


One flaw in the statistical analysis above is that the number of LEP and lEP students 
is much lower than that of the general population, slightly reducing the validity of 
the tests; however, the differences in mean were dramatic enough to conclude that 
these students should not be considered in the general analysis. If the amount of 
LEP or lEP students were lopsided in any group, the analysis would be 
compromised, thus lEP and LEP students are excluded from all following analyses. 

Grade Level 

Students may perform differently on the MAP tests depending upon which grade 
level they are. The MAP tests are not universally normed. Thus analyses of variance 
were run to see if this demographic would need to be added as a controlled variable. 
When all grades for which MAP data were available (grades 3 through 12] were 
considered, significant differences were found for every MAP test at p < .001, with F- 
values ranging from 311.6 (df = 6, 3401) to 1246.8 [df = 6, 11036). When grades 3 
through 8 were isolated for analysis, there was still a significant difference across 
grades for every test, with F-values ranging from 482.0 (df = 2, 2969) and 1168.1 (df 
= 5, 10571). All trends indicated that higher grades were receiving higher MAP 
scores. Thus grade levels were separated, with all subsequent analyses being run 
independently on each. 

Gender 

The effect of gender was analyzed by looking for differences in MAP performance 
between males and females. Focusing on grade 3 through 8, a significant difference 
was found on Communication Arts for every grade at p < .001, with females scoring 
higher each time. A significant difference was also found in Mathematics in grade 5, 
with males scoring significantly higher, t(1894) = 3.0, p < .01. Males also scored 
higher in Social Studies in grades 4 and 8, t(1393) = 2.3, p < .05, and t(1574) = 3.1, p 
< .01, respectively, and Science in grade 7, t(1531) = 3.1, p < .01. These results 
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indicate enough gender differences to include it as a control variable in more- 
complex analyses. 

Race 

The effect of race was analyzed by looking for differences in MAP performance 
across students who are African American, Asian American, Caucasian, Hispanic, 
and Native American/Alaska Native. Focusing on grades 3 through 8, every MAP test 
in every grade showed significant differences across race at p < .01, except Science 
in grade 8, due to lack of diversity; however, the validity is lowered due to the varied 
numbers of students in each race, with Caucasian, then African American, having 
significantly more students. This analysis still shows that the effect of each race 
should be independently controlled for in subsequent and more advanced analyses. 
Exact effects are not reported here since each race will be considered independently 
for future analyses. 

FRL 

The effect of free and reduced lunch status (FRL) was analyzed by looking for 
differences in MAP performance between FRL and non-FRL students. Focusing on 
grades 3 through 8, every MAP test in every grade showed significant differences 
across FRL at p < .001, except grade 8 Science, which was significant at p < .01. 

These results indicate clearly that FRL should be used as a control variable in more 
complex analyses. 

2006 MAP scores 

Often the best predictor of a student's score on an academic test is his or her 
previous score. Thus the relationship between students' 2006 MAP scores and 2007 
MAP scores was analyzed. There were only enough matched students across the two 
years to look at Communication Arts (CA) and Mathematics (MA), but both showed 
significant correlations between the 2006 and 2007 scores, r[5029) = .733, p < .001 
and r(5032) = .793, p < .001, respectively. These results indicate that students' 2006 
scores should be used as a control variable in more complex analyses. 

RELATIONSHIP BETWEEN STUDENT SCORES AND TEACHER 
MASTERY 

The first analysis looked at the relationship between student MAP scores and 
Teacher Mastery data. Since teachers have immediate contact with students, it is 
possible that the strongest effects on MAP scores will be seen in what teachers learn, 
which is in part measured by Teacher Mastery scores. Fidelity was shown to link to 
Teacher Mastery; perhaps stronger Teacher Mastery will then lead to teaching 
practices that result in better student MAP scores. 
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Correlation analysis was the first step in analyzing this relationship. Correlations 
were run for each grade looking at the relationship of each student MAP test to each 
Teacher Mastery artifact [see Table 25). Results indicated that students’ MAP 
performance was positively correlated to better Teacher Mastery scores. Not every 
item was correlated, but every significant relationship found was in the positive 
direction. Positive correlations were found in grades 3, 4, and 7 for student MAP 
scores and their teachers' Lesson Plan. Positive relationships were also found in 
grades 3 and 7 for WebQuests and in grades 4, 5, and 7 for Classroom Websites. 


Table 25: Correlations between student MAP tests and Teacher Mastery 
artifacts 



Teacher Mastery Item 

Grade 

MAP test 

Lesson Plan 

WebQuest 

Classroom Website 

3 

CA 

24* ♦ * 

ns 

ns 

3 

MA 

25*** 

.14** 

ns 

3 

SC 

ns 

ns 

ns 

4 

CA 

ns 

ns 

.10, p=.06 

4 

MA 

ns 

ns 

.10, p=.06 

4 

SS 

.20** 

ns 

25** 

5 

CA 

ns 

ns 

* * 

5 

MA 

ns 

ns 

22*** 

6 

CA 

ns 

ns 

ns 

6 

MA 

ns 

ns 

ns 

7 

CA 

2'7* * * 

.14* 

22*** 

7 

MA 

.18* 

.16* 

22*** 

7 

SC 

ns 

ns 

ns 

8 

CA 

ns 

ns 

ns 

8 

MA 

ns 

ns 

ns 

8 

SC 

ns 

ns 

ns 

8 

SS 

ns 

ns 

ns 


*p < .05, **p < .01 ***p<.001 ns - not significant 


Overall, these correlations show a strong positive relationship between the artifacts 
teachers are making during eMlNTS and how their students are performing on the 
MAP test. Higher teacher Lesson Plan ratings were consistently related to higher 
student MAP scores, in Communication Arts, Mathematics, and Social Studies. To 
further explore this relationship, regression analyses were performed, controlling 
for covariates, to analyze the impact of Teacher Mastery on MAP scores. The first 
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step of the regression controlled for the largest covariate, students' 2006 MAP 
scores. Step 2 added the other covariates to the regression [race, gender, FRL) and 
step 3 added each Teacher Mastery score independently to see if they explained a 
significant amount of variance to the regression after controlling for each covariate. 

Teachers’ Lesson Plan ratings repeatedly explained variance in students’ MAP 
scores on both Communication Arts (CA) and Mathematics (MA) (see Table 26). 
Except for grade 6 CA, increased scores on Lesson Plans predicted increased MAP 
scores for every significant finding. The anomaly of grade 6 CA can mostly be 
attributed to an unreliable analysis, due to the disproportionately small number of 
grade 6 students taking this assessment. These results, taken with the correlations 
above, indicate that students whose teachers can create a good Lesson Plan [that 
incorporates the eMlNTS concepts] perform better on MAP assessments. 


Table 26: Amount of variance in students' MAP data explained by teachers' 
Lesson Plans, after controlling for all covariates. 


Grade 

MAP test 

Variance explained by 
Lesson Plan 

Beta 

t 

P 

3 

CA 

2.0%* 

.14 

3.46 

p < .001 

3 

MA 

2.2%* 

.15 

3.71 

p < .001 

4 

CA 

ns 




4 

MA 

ns 




5 

CA 

0.5% 

.07 

2.10 

p< .05 

5 

MA 

ns 




6 

CA 

3.5%** 

-.19 

-3.15 

p< .01 

6 

MA 

ns 




7 

CA 

2.4% 

.16 

3.48 

p < .001 

7 

MA 

ns 




8 

CA 

1.6% 

.13 

2.37 

p< .05 

8 

MA 

0.8% 

.09 

1.86 

p=.065 


*Analyses were run without the 2006 MAP scores in the model because there were no scores for grade 2. 
**This may be attributed to unreliable analysis due to the disproportionately small n for grade 6. 


Teachers’ WebQuest ratings were a less reliable predictor of student MAP scores, 
but positive trends can still be seen. Table 25 above shows three positive 
correlations for WebQuests, indicating that students whose teachers can create 
better WebQuests perform better on MAP assessments in CA and MA. The 
regression analyses provided more mixed results (see Table 27). The grade 6 CA 
assessment was again a negative anomaly, and the other negative result, grade 4 
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MA, explained less than 1% of the variance. The grade 5 MA is a reliable indicator^ 
and it points to a positive relationship between better WebQuest creation and 
higher MAP performance. 


Table 27: Amount of variance in students' MAP data explained by teachers' 
WebQuests, after controlling for all covariates. 


Grade 

MAP test 

Variance explained by 
WebQuest 

Beta 

t 

P 

3 

CA 

ns* 




3 

MA 

ns* 




4 

CA 

ns 




4 

MA 

0.9% 

-.10 

-3.03 

p< .01 

5 

CA 

ns 




5 

MA 

3.3% 

.19 

3.68 

p < .001 

6 

CA 

1.9%** 

-.14 

-2.29 

p< .05 

6 

MA 

ns 




7 

CA 

ns 




7 

MA 

ns 




8 

CA 

ns 




8 

MA 

ns 





*Analyses were run without the 2006 MAP scores in the model because there were no scores for grade 2. 
**This may be attributed to unreliable analysis due to the disproportionately small n for grade 6. 


Teachers’ Classroom Websites are another strong, positive predictor of how 
students would perform on their MAP assessments. Preliminary analysis showed 
seven positive correlations for Classroom Websites (see Table 25 above), indicating 
that students with teachers who are able to create a better Classroom Website 
perform better on MAP assessments in CA, MA, and SS. The regression analysis (see 
Table 28) also showed positive results. In grades 5 and 7, higher Classroom Website 
scores predicted higher student MAP scores. 
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Table 28: Amount of variance in students' MAP data explained by teachers' 
Classroom Websites, after controlling for all covariates 


Grade 

MAP test 

Variance explained by 
Classroom Website 

Beta 

t 

P 

3 

CA 

ns* 




3 

MA 

ns* 




4 

CA 

ns 




4 

MA 

ns 




5 

CA 

4.0% 

.22 

3.68 

p < .001 

5 

MA 

1.7% 

.14 

2.57 

p< .02 

6 

CA 

ns 




6 

MA 

ns 




7 

CA 

1.8% 

.14 

2.99 

p< .01 

7 

MA 

ns 




8 

CA 

ns 




8 

MA 

ns 





*Analyses were run without the 2006 MAP scores in the model because there were no scores for grade 2. 


Analyses were also run within districts with a large enough number of students to 
look for positive district-by-district trends of Teacher Mastery and students’ MAP 
scores. Several positive relationships were found. In one district, grade 3 CA and MA 
were positively correlated with Lesson Plan, r(132] = .21, p < .02 and r(132) = .19, p 
< .05, respectively. In another, grade 3 CA was marginally correlated (because of the 
low n] with Lesson Plan, r(42) = .27, p=.08. In a third district, grade 4 SS was 
positively correlated with Lesson Plan, r(112) = .21, p < .05; and grade 4 CA was 
marginally correlated to WebQuest, r[94) = .19, p = .07. In a fourth district, grade 5 
MA was positively correlated with WebQuest, r[52) = .39, p < .01. In a fifth district, 
grade 7 CA and MA scores were positively correlated with Lesson Plan, WebQuest, 
and Classroom Website at p < .05. Grade 8 CA and MA were also positively 
correlated to Lesson Plan and Classroom Website. A few negative relationships were 
found, which were far outweighed by the positive findings; however, this did 
indicate that there were some district-by-district variations in relationships among 
student achievement and Teacher Mastery that cannot be fully accounted for by 
these analyses. 
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RELATIONSHIP BETWEEN STUDENT SCORES AND PD FIDELITY 


PD Fidelity was positively related to Teacher Mastery scores; thus it wa possible 
that PD Fidelity had a smalh but significant, impact on student achievement. Even if 
not reflected in Teacher Mastery scores, teachers may have learned certain core 
concepts from sessions with strong fidelity that would lead to improved pedagogical 
practices and thus to improved student scores. Correlations were run to detect any 
relationships between MAP performance and PD Fidelity scores from eMINTS PD 
sessions. Correlations were run with the overall PD Fidelity measure created from 
the Program Construct factors and checked with the overall PD Fidelity measure 
created from the Factor Analysis-Based factors. 

Positive relationships were found [see Table 29). Results indicate that as overall 
Fidelity of increased, student scores were higher. No negative correlations were 
found. 

Table 29: Correlations between student MAP scores and PD Fidelity of 
eMINTS professional development 


Grade 

MAP test 

Overall Fidelity 

3 

CA 

ns 

3 

MA 

o 

* 

3 

SC 

ns 

4 

CA 

* * 

4 

MA 

2Q* * * 

4 

SS 

22** 

5 

CA 

20* * * 

5 

MA 

2Q* * * 

6 

CA 

ns 

6 

MA 

ns 

7 

CA 

ns 

7 

MA 

ns 

7 

sc 

ns 

8 

CA 

.19** 

8 

MA 

.15*(p=.08) 

8 

SC 

ns 

8 

SS 

ns 


*p < .05 **p<.01 ***p < .001 

*Only significant with fidelity score created from the Factor Analysis-Based factors. 
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To further explore this relationship, regression analyses were performed that 
controlled for all covariates (see Table 30). The results confirmed the positive 
association between MAP scores and overall PD Fidelity, especially with teachers of 
grades 5 and 8. The other results are mixed and may mean the covariates are so 
strong that the analysis is not showing the relationship. 


Table 30: Amount of variance in students' MAP data explained by PD 
Fidelity, after controlling for all covariates 


Grade 

MAP test 

Variance explained by PD 
Fidelity 

Beta 

t 

P 

3 

CA 

ns* 




3 

MA 

ns* 




4 

CA 

ns 




4 

MA 

ns 




5 

CA 

1.3% 

.13 

3.7 

p < .001 

5 

MA 

0.9% 

.10 

3.1 

p< .01 

6 

CA 

2.7% 

-.17 

-2.7 

p< .01 

6 

MA 

ns 




7 

CA 

ns 




7 

MA 

ns 




8 

CA 

2.2% 

.15 

2.7 

p< .01 

8 

MA 

2.9% 

.18 

3.4 

p < .001 


*Analyses were run without the 2006 MAP scores in the model because there were no scores for grade 2. 


The positive results above prompted further analyses that looked at the specific 
aspects of PD Fidelity and which were correlating to improved student outcomes 
(see Table 31). A few negative correlations were found, but these were generally 
mixed with multiple positive correlations for the same student MAP test. This may 
indicate that the instructional specialists whose teachers had students with better 
MAP scores were simply focusing on certain elements of PD Fidelity more often 
(those with the positive correlations). Generally, the results were positive. 
Significant positive relationships were found on CA, MA, and SS across every grade 
except 6* (probably due to the lower n). Every factor of fidelity also had at least one 
positive relationship to student MAP scores. Only doing Structured Activities had 
more negative correlations than positive. The other activities may be more 
beneficial to helping teachers in the classroom, especially in grades 3 to 5. 
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Table 31: Correlations between student MAP scores and eMINTS PD Fidelity 
factors. 

Dark gray areas indicate positive correlations and light gray indicate negative correlations. 

(Legend-. SA=Structured Activities, PD=Participant-Led Discussion, SI=Scaffolding Instruction, 
FD=Facilitating Discussion, AW=Active Work/Learning, MI=Modeling Instruction, CB=Community Building, 
TU=Technology Utilization, CP=Connection to Practice, IBL=lnquiry Based Learning.) 




Factor Analysis-Based Factors 

Program Construct Factors 

Grade 

MAP 

SA 

PD 

SI 

FD 

AW 

Ml 

CB 

TU 

CP 

IBL 

3 

CA 

ns 

ns 

22** 

ns 

ns 

ns 

ns 

.08* 

ns 

ns 

3 

MA 

-.09* 

.09* 

20*** 

ns 

ns 

.09* 

ns 

20* * * 

ns 

ns 

3 

SC 

ns 

ns 

ns 

ns 

ns 

ns 

ns 

ns 

ns 

ns 

4 

CA 

-.13 

.12* 

23*** 

.09* 

ns 

20* * * 

ns 

22*** 

.11* 

.11* 

4 

MA 

-.10* 

.14** 

22*** 

.12* 

ns 

2Q* * * 

.10* 

22*** 

14** 

.11* 

4 

SS 

ns 

2Q* * * 

29** 

.13* 

22** 

24* * * 

23*** 

.16* 

.17* 

.18* 

5 

CA 

ns 

.11* 

32*** 

20* * * 

ns 

22* * * 

.10* 

32*** 

2Q* * * 

2g* * * 

5 

MA 

ns 

22* ♦ * 

30*** 

22*** 

ns 

22* * * 

22* * * 

33*** 

22*** 

3 2* * * 

6 

CA 

ns 

ns 

ns 

ns 

ns 

ns 

ns 

ns 

ns 

ns 

6 

MA 

ns 

ns 

ns 

ns 

ns 

ns 

ns 

ns 

ns 

ns 

7 

CA 

ns 

.15* 

42*** 

ns 

ns 

2g* * * 

ns 

23*** 

-.18* 

ns 

7 

MA 

ns 

23*** 

35*** 

ns 

ns 

22* * * 

22** 

22*** 

-.15* 

.13* 

7 

SC 

ns 

ns 

ns 

ns 

ns 

ns 

ns 

ns 

ns 

ns 

8 

CA 

25** 

ns 

-.16* 

.19* 

ns 

ns 

.20* 

ns 

ns 

29* * * 

8 

MA 

22** 

ns 

ns 

.15* 

ns 

ns 

.18* 

ns 

ns 

25** 

8 

SC 

- 28** 

ns 

ns 

ns 

ns 

ns 

- 28** 

ns 

- 28** 

ns 

8 

SS 

ns 

ns 

ns 

ns 

ns 

ns 

ns 

ns 

ns 

ns 


*p < .05 **p<.01 ***p<.001, 

*p < .08 


Table 31 shows that when there is a significant positive relationship of one factor to 
any given MAP scores, there is usually a positive relationship with at least one other 
factor of PD Fidelity to the same MAP assessment. This is especially true when 
looking at grades 4 and 5, but grades 3, 7 and 8 also show this pattern. This may 
indicate that instructional specialists who are doing one aspect of the professional 
development well are also doing other aspects well, and these are the instructional 
specialists whose teachers have students with higher MAP scores. Thus it may not 
be one individual component of PD Fidelity that helps teachers boost students’ MAP 
scores, but is more likely a combination of many factors. 

Individual regressions were not run on individual aspects of PD Fidelity, for this 
reason and because of concern with alpha inflation and the mixed pattern of results 
from the overall regression. District-by-district analyses were also foregone, due to 
the lack of variability of PD Fidelity data within districts, since each district has only 
one or a few instructional specialists. 
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RELATIONSHIP BETWEEN STUDENT SCORES AND CLASSROOM 
VISITS 


Aspects of Classroom Visits were positively related to PD Fidelity scores and to 
Teacher Mastery scores. Thus it was possible that Classroom Visits would show 
small but significant relationships to student achievement. Teachers who ask their 
instructional specialists for specific kinds of support during Classroom Visits may be 
building certain skills and knowledge that would lead to improved pedagogical 
practices and, as a result, higher student scores. An analysis was run to detect any 
relationships between MAP performance and Classroom Visits (see Table 32). 

Table 32: Correlations between student MAP scores and Classroom Visits 

Included variables are the total number of visits, and percent time doing each activity. Light gray boxes 
indicate a negative correlation and dark gray boxes indicate a positive correlation. 


Grade 

MAP 

Number 
of visits 

Modeling 

Instructions 

Lesson 

Pianning 

Technoiogy 

Assistance 

Reflective 

Practice 

Problem 

Solving 

other 

3 

CA 

ns 

ns 

ns 

ns 

ns 

ns 

ns 

3 

MA 

-.08* 

-.09* 

ns 

ns 

ns 

ns 

ns 

3 

SC 

ns 

ns 

ns 

-.12* 

ns 

ns 

ns 

4 

CA 

-.18* 

ns 

.11* 

ns 

ns 

ns 

ns 

4 

MA 

-.18* 

-.09* 

.11* 

ns 

ns 

ns 

ns 

4 

SS 

ns 

-.12* 

ns 

-.12* 

25*** 

ns 

ns 

5 

CA 

ns 

22*** 

* * 

ns 

ns 

ns 

.10* 

5 

MA 

ns 

20* * * 


ns 

ns 

b 

tD 

* 

.10* 

6 

CA 

ns 

ns 

ns 

ns 

ns 

ns 

ns 

6 

MA 

ns 

ns 

ns 

ns 

ns 

ns 

ns 

7 

CA 

19** 

ns 

ns 

22*** 

- 20** 

24* * * 

4j^*** 

7 

MA 

ns 

ns 

ns 

- 20** 

-.18* 

-.15* 

22* * * 

7 

SC 

ns 

ns 

-.49* 

ns 

ns 

52* * 

ns 

8 

CA 

ns 

ns 

22*** 

22** 

ns 

30*** 

ns 

8 

MA 

ns 

ns 

22*** 

-.24** 

ns 

29*** 

ns 

8 

SC 

ns 

ns 

ns 

ns 

ns 

ns 

ns 

8 

SS 

ns 

ns 

ns 

ns 

ns 

ns 

ns 


*p < .05 **p<.01 ***p<.001, 

*p < .06 
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The correlation analysis showed mixed findings. Generally, higher student MAP 
scores were associated with more time on Lesson Planning and less time on 
Modeling Instruction, Technology Assistance, and Problem Solving. This may 
indicate that teachers who spend time creating and refining their lessons with their 
instructional specialists provide better instruction for their students, which may 
then boost student scores. Conversely, teachers who request that their instructional 
specialists model instruction, provide technology assistance, and solve problems 
may be struggling more with the eMINTS practices or with integrating technology in 
general, and therefore may not be having as strong an impact on their students’ 

MAP scores. Results for number of visits and reflection were mixed and 
inconclusive. Further exploratory research would be needed to understand these 
effects. The positive relationships found with 'other activities’ indicate a need for 
more research about what instructional specialists are doing during their Classroom 
Visits that is not captured on the Classroom Visit Record instrument that may have 
positive effects on teacher practices and student test scores. 

Individual regressions were not run on individual aspects of Classroom Visits 
because of the mixed results, covariance, and concern with alpha inflation. District- 
by-district analyses were performed. The alpha inflation is a limitation to this 
analysis; however, the district analyses corroborated some of the overall findings, 
but also uncovered a few unique relationships that emphasize the need for further 
exploration of district practices. Table 33 shows correlations for districts whose 
results corresponded to the overall findings, and Table 34 shows correlations whose 
results did not correspond. 
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Table 33: Correlations between student MAP scores and Classroom Visits 

Broken up by districts that correspond to the overall findings. Included variables are the total number of 
visits and percent time doing each activity. Light gray boxes indicate a negative correlation and dark gray 
boxes indicate a positive correlation. 


Grade 

MAP 

Number 
of visits 

Modeling 

Instructions 

Lesson 

Planning 

Technology 

Assistance 

Reflective 

Practice 

Problem 

Solving 

other 

3 

CA 

ns 

ns 

ns 

ns 

ns 

ns 

ns 

3 

MA 

-.20* 

ns 

ns 

ns 

ns 

ns 

ns 

3 

SC 

ns 

ns 

ns 

-.16* 

ns 

ns 

ns 

4 

CA 

ns 

ns 

ns 

ns 

ns 

ns 

ns 

4 

MA 

ns 

-.39** 

-.30*) 

ns 

ns 

ns 

ns 

ns 

4 

SS 

ns 

-.29* 

ns 

ns 

31** 

.26^ 

ns 

ns 

5 

CA 

ns 

-.19* 

* * 

ns 

ns 

ns 

ns 

5 

MA 

ns 

-.19* 

.21* 

.60*** 

ns 

ns 

22** 

-.30* 

ns 

6 

CA 

ns 

ns 

ns 

ns 

ns 

ns 

ns 

6 

MA 

ns 

ns 

ns 

ns 

ns 

ns 

ns 

7 

CA 

* * 

ns 

ns 

-.23* 

-.23* 

-.23* 

52*** 

2Q* * * 

7 

MA 

ns 

ns 

ns 

-.19* 

-.19* 

-.19* 

44* * * 

.68*** 

7 

SC 

ns 

ns 

ns 

ns 

ns 

ns 

ns 

8 

CA 

ns 

ns 

.23* 

ns 

ns 

-.23 

ns 

8 

MA 

ns 

ns 

.22* 

ns 

ns 

-.21 

-.30* 

ns 

8 

SC 

ns 

ns 

ns 

ns 

ns 

ns 

ns 

8 

SS 

ns 

ns 

ns 

ns 

ns 

ns 

ns 


*p < .05 **p<.01 ***p<.001 ns - not significant 
*p < .07 

These results show that many correlations that were found overall were also found 
within districts. This again shows the positive effect of working on Lesson Plans and 
the negative effect of having to spend extra time on modeling instruction and 
problem solving. The effect of having more visits is again mixed. Correlations that do 
not overlap with the overall findings provided some mixed results Table 34 shows 
spending time on Lesson Planning is related to higher student scores, although 
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results are slightly mixed. Table 34 also shows that other activities can be related to 
lower student scores in some districts, emphasizing the need for further research 
about what is happening. Results for Technology Assistance are again strongly 
negative. Modeling Instruction shows mixed results. 

Table 34: Correlations between student MAP scores and Classroom Visits 

Broken up by districts that do not correspond to the overall findings. Included variables are the total number of visits, 
and percent time doing each activity. Light gray boxes indicate a negative correlation and dark gray boxes indicate a 
positive correlation. 


Grade 

MAP 

Number of 
visits 

Modeling 

Instructions 

Lesson 

Planning 

Technology 

Assistance 

Reflective 

Practice 

Problem 

Solving 

Othe 

r 

3 

CA 

-.18* 

.20* 

-.19* 

.18* 

-.29* 

ns 

ns 

.19* 

3 

MA 

ns 

.36* 

.26** 

-.28* 

.18* 
28* * * 

-.35* 

ns 

3 

SC 

.30* 

-.26* 

.29* 

ns 

ns 

ns 

ns 

4 

CA 

ns 

-.29* 

-.32* 

ns 

-.26* (H) 

.30* 

ns 

.28* 

4 

MA 

ns 

ns 

ns 

ns 

.31* 

.28* 

.36** 

-.26* 

4 

SS 

.22* 

ns 

37* * 
.27* 

ns 

ns 

-.35** 

.27* 

-.if 

5 

CA 

37* * * 

ns 

ns 

ns 

ns 

ns 

ns 

5 

MA 

-.26** 

-.30* 

ns 

ns 

- 60*** 

58* * * 

ns 

ns 

6 

CA 

ns 

ns 

ns 

ns 

ns 

ns 

ns 

6 

MA 

ns 

ns 

ns 

ns 

ns 

ns 

ns 

7 

CA 

ns 

.23* 

.50*** 

.23* 
54* * * 

ns 

ns 

ns 

ns 

7 

MA 


.19* 

* * 

.19* 
54* * * 

ns 

ns 

ns 

ns 

7 

SC 

ns 

ns 

ns 

ns 

ns 

ns 

ns 

8 

CA 

.23* 

-.23 

ns 

.23* 

.23* 

ns 

ns 

8 

MA 

.21* 

-.21 

ns 

.21* 

.21* 

.27* 

ns 

ns 

8 

SC 

ns 

ns 

ns 

ns 

ns 

ns 

ns 

8 

SS 

ns 

ns 

ns 

ns 

ns 

ns 

ns 


*p < .05 **p<.01 ***p<.001, 
p < .07 

Overall, the results show a great deal of variation across districts. This variation of 
relationship between Classroom Visits and student performance could be due to the 
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level of ability of students in each district, the teacher quality in each district, the 
instructional specialists and how they approach eMINTS and the Classroom Visits, 
as well as any interaction among these factors. These results, on top of the 
demographic differences between districts, show a limit to this analysis, but many of 
the findings indicate that teachers’ use of instructional specialists' Classroom Visits 
may affect later student performance. 

RELATIONSHIP BETWEEN STUDENT SCORES AND EMINTS STATUS 

Analysis of program effects on student scores is not appropriate for this study. To 
show impact on student scores based on program, the student sample for each 
program would need to be matched through planned methodology, or at least have 
demographic information that is similar, with slight differences being statistically 
controlled. This would be done through an experimental or quasi-experimental 
design. The current study was not designed to create such populations of students. 

Prior evaluation of eMINTS has utilized a quasi-experimental design in order to test 
program impact. This was accomplished by using comparison classrooms within 
schools to represent students with comparable demographics. By having 
comparison non-eMlNTS teachers for eMINTS teachers within a school and grade 
level, groups of equal size and demographics from each district would be formed, 
allowing for causal analysis to be valid across the entire population of eMINTS and 
non-eMlNTS students. 

This study was designed to be an exploratory research project aiming to discover 
the relationships among Program Fidelity, Teacher Mastery, and student 
achievement. Student impact was the final concern; however, how student impact 
varied by program [eMINTS Comp PD, eMINTS 4ALL, and non-eMINTS) was not 
taken into consideration. This year many of the smaller districts and schools and 
grade levels have become all or majority eMINTS and could not provide equivalent 
classroom groups for comparison. 

The addition of two different programs, eMINTS Comprehensive and eMINTS 4ALL, 
made the creation of comparison groups even more difficult. The design would have 
to provide equal groups across all three categories across ah districts. This would 
require each district to implement both programs equally across grade level, or have 
a comparison district in which the demographics and effects would be comparable. 
Table 35 shows that the number of students in each program category varies 
substantially across grade levels. Districts did not have equivalent numbers of 
teachers and students in each program. This is shown in Table 36 by the disparity of 
program implementation across districts at the grade 4 level. This shows that the 
programs have different district composites. Optimally, the data should be balanced 
across districts for valid comparison across programs. Our analysis so far has 
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unveiled numerous district differences across program implementation strategies, 
effects, and component relationships. This district imbalance leads to program 
groups where comparison would not be valid. 

Table 35: Number of Communication Arts MAP scores for each grade, 
across programs. 


Grade 

4ALL 

Comp PD 

Non-eMINTS 

eMINTS 

Veterans 

3 

234 

491 

963 

12 

4 

84 

499 

1022 

101 

5 

467 

274 

952 

19 

6 

497 

97 

782 

47 

7 

528 

222 

736 

0 

8 

545 

518 

840 

0 


Table 36: Number of Communication Arts MAP scores for grade 4, across 
districts and programs 


District 

4ALL 

Comp PD 

Non-eMINTS 

eMINTS 

Veterans 

Carrollton 

16 

0 

55 

0 

Chillicothe 

68 

0 

20 

22 

Francis Howell 

0 

131 

947 

21 

Gasconade 

0 

60 

0 

0 

Hartville 

0 

46 

0 

0 

Jefferson City 

0 

21 

0 

20 

North Kansas 

0 

67 

0 

0 

Perry County 

0 

82 

0 

38 

Sparta 

0 

41 

0 

0 

Webster Grove 

0 

51 

0 

0 


Furthermore, not all districts could even provide students from both programs in 
each grade. For example, Carrollton had eMlNTS Comprehensive teachers in grades 
3, 5, and 7, but only eMlNTS4ALL teachers in grades 4, 6, and 8 (see Table 37). 
Carrollton also only had comparison rosters for the teachers from the 4ALL 
program. Thus the larger districts, mostly Francis Howell, provided the 
overwhelming majority of the control students (see Table 38). 
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Table 37: The number of Communication Arts MAP scores for each grade, 
across programs, for the Carrollton School District. 


Grade 

4ALL 

Comp PD 

Non-eMINTS 

eMINTS 

Veterans 

3 

0 

62 

0 

0 

4 

16 

0 

55 

0 

5 

0 

65 

0 

0 

6 

17 

0 

52 

0 

7 

0 

66 

0 

0 

8 

24 

0 

40 

0 


Table 38: Number of Communication Arts MAP scores for each grade, 
across programs, for the Francis Howell School District 


Grade 

4ALL 

Comp PD 

Non-eMINTS 

eMINTS 

Veterans 

3 

0 

180 

913 

0 

4 

0 

131 

947 

21 

5 

224 

22 

876 

2 

6 

434 

0 

730 

0 

7 

451 

0 

725 

0 

8 

489 

14 

791 

0 


To compare students grouped by program would require comparing student groups 
that are unbalanced by grade and district. Considering the purpose of the study, the 
district differences previously discussed, and the lack of valid comparison groups in 
most districts, comparative program analysis would not be valid and thus was not 
performed. 
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Section V: Discussion 


Throughout the nine-year history of eMINTS, the eMINTS program staff have 
demonstrated a strong commitment to evaluation; they have been using evaluation 
not only to show impact but also to learn about and strengthen the program. This 
evaluation has generated research tools and findings that can contribute to the 
ongoing process of reflection and learning that the eMINTS program staff have 
consistently undergone. 

EDC/CCT’s evaluation was designed to provide the eMINTS program staff with a 
variety of instruments and procedures that can be used in the future for internal 
evaluation of the program as it scales up, and for potential future studies of the 
program. Some of the instruments and procedures produced through this evaluation 
include: 

• PD Fidelity Snapshot and Checklist Observation Instruments. These two instruments 
look at how well a professional development session addresses the core eMINTS 
constructs. They can be used in a number of ways. Not only could they be used again 
to measure PD Fidelity in a future study of eMINTS program impact, they could also 
be used by the regional specialists or program administrators in the new expansion 
programs to assess professional development quality, and as a tool for PD4ETS self- 
assessment. 

• Classroom Visit Record Instrument. This instrument provides a consistent and simple 
record of what instructional specialists do to support teachers during Classroom 
Visits. This instrument could also be used in further studies of the program. In 
addition, program staff could review data from these records to see what kinds of 
activities are occurring most often during Classroom Visits, and instructional 
specialists could use them to set goals with their teacher participants about the kind 
of support they want to experience and to reflect on whether those goals are being 
met. 

• Teacher Mastery Rubrics. These instruments allow a reviewer to assess how well 
teacher artifacts are aligned with the core eMINTS concepts. Again, these could be 
used in a future study of the program. They can also be used by regional or central 
program staff to assess teacher portfolio quality for certification purposes, and they 
can be used as guides for teacher participants to understand what constitutes high- 
quality Lesson Plans, WebQuests and Classroom Websites. 

In addition to the instruments, this evaluation has produced a number of findings 
that provide information about the fidelity of eMINTS professional development and 
Teachers’ Mastery of eMINTS concepts. This information can, first, serve as a 
baseline against which to compare other programs as eMINTS scales up to other 
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states and countries. Before these data about Program Fidelity and Teacher Mastery 
were collected and analyzed, program quality and teacher impact could primarily be 
assessed only by those with a great deal of expertise and background with the 
program. The analyses in this report now enable the eMlNTS program staff to 
provide those who are introducing the program in a new setting with clear, 
quantifiable expectations for Program Fidelity and Teacher Mastery. 

The findings from the analyses of Program Fidelity and Teacher Mastery also 
highlight where the program is strong and where it may need to be improved. The 
PD Fidelity data from both the Checklist and the Snapshot instruments indicated 
that the instructional specialists are generally very faithful in delivering the 
professional development in an engaging manner that has participants take part in 
hands-on collaborative group work. Instructional specialists also appeared to 
structure the professional development appropriately, by introducing the session 
essential question and goals and reviewing these at the end. They also model some, 
but not all, of the key elements of Inquiry-Based Learning, such as having 
participants engage in independent research and analyze and synthesize what they 
learn. However, although instructional specialists were modeling effective 
instruction, they did not seem to be as effective at engaging the participants in 
explicit discussions of how to integrate core eMINTS concepts, such as Community 
Building and Inquiry-Based Learning, into instruction. The data from the Teacher 
Mastery assessment also suggest that teachers need extra support in Inquiry-Based 
Learning strategies, as well as in tailoring their instruction to accommodate student 
diversity. 

The analyses of the combined datasets also offered some interesting information 
about how the program inputs were associated with teacher and student outcomes. 
One striking set of associations was the one between Classroom Visits and teacher 
and student outcomes. The more time spent during Classroom Visits on Lesson 
Planning, the greater the impact on both Teacher Mastery of the Lesson Plan and on 
student achievement. It is difficult to say whether this is causal, or if those teachers 
who choose to use the Classroom Visit time in this way also happen to be more 
effective teachers, but this finding does provide some insight into one possible "best 
practice" for Classroom Visits. 

Another important finding from the combined analysis is how strong the 
associations were, at least for some grades, between PD Fidelity and student 
achievement. It is particularly interesting to note where this relationship exists even 
when there are not strong correlations between Teacher Mastery and student 
achievement. This suggests that high-quality professional development may be 
having an impact on teachers and their instruction that is not adequately captured 
in their portfolio artifacts. 


eMINTS Program Evaluation Report 


73 


It is important to keep the findings in perspective by noting the limitations of this 
study. One of the main limitations of this evaluation was that a limited amount of 
data was collected from the professional development sessions. In most cases, 
instructional specialists were observed only once, though some instructional 
specialists were observed twice. In our analyses, this limited data had to represent 
the fidelity of all professional development experienced by teachers. While practical 
realities of time and staffing made it impossible for this study to observe more 
professional development sessions, future studies of Program Fidelity would be 
more robust if evaluators observed more professional development sessions per 
instructional specialist and averaged the scores for each item across sessions. 

Another limitation of the study, similar to the problem with PD Fidelity, is that 
Teacher Mastery was constructed, again, from a limited set of data — only the Lesson 
Plan for eMINTS4ALL teachers, and the Lesson Plan, WebQuest, and Classroom 
Website for eMINTS Comprehensive teachers. A better measure of Teacher Mastery 
would come from classroom observations using instruments comparable to those 
created for measuring PD Fidelity. However, conducting an adequate number of 
observations to fully measure a teacher's instructional practices was beyond the 
scope of this evaluation; it would be a considerable undertaking. 

A third, but less problematic, limitation was in the design of the Classroom Visit 
Record. Unlike the previous two instruments, this instrument produced a great deal 
of data about what instructional specialists did with teachers over a number of 
months. However, our analyses of the combined Classroom Visit and student MAP 
data revealed that engaging in "Other" activities was positively correlated with 
student MAP scores in grades 5 and 7. This suggests that there is some additional 
Classroom Visit activity that is not captured by the instrument that may be a very 
effective for teachers. It may be useful to conduct interviews with instructional 
specialists to find out what they are doing with teachers that they would identify on 
the record as "Other" and include these activities in a revised Classroom Visit 
Record Instrument. 

A number of next steps follow logically from this evaluation. The first, already in the 
design of this evaluation, is to conduct a series of combined analyses with the 
Program Fidelity and Teacher Mastery data and a second year of MAP data. We will 
analyze the 2008 MAP data from the students in this year’s study, to see if the 
program continues to have an impact on those students, and the 2008 MAP data 
from the new students of teachers in this year’s study, to determine if the program 
continues to have an impact on those teachers and the kind of instruction they 
provide. This analysis will serve as a validity check on the findings of the current 
evaluation, and will also provide information about the persistence of program 
impact. 
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Another logical next step would be to conduct a similar evaluation in some of the 
new states where eMINTS is being implemented. This would be an important means 
of ensuring that the program maintains its quality and impact as it scales up, and of 
validating the findings from this evaluation. 

The third next step would be to seek funding for a randomized control trial to 
measure program impact on students and teachers. Although, as mentioned above, a 
number of evaluations have provided evidence of positive program impact on 
students, a randomized control trial would provide a stronger, causal case for 
program impact, should similar results be found. The fidelity instruments designed 
for this study could play an integral role in a randomized control trial, because they 
would enable researchers to include information about Program Fidelity as a key 
variable in the analysis. 

Over the past nine years, the evaluations of the eMINTS program have generated a 
wealth of important findings about the program. A strong and consistent body of 
evidence, produced through quasi-experimental studies, shows the positive impact 
of eMINTS on teacher and student learning. This new evaluation provides another 
lens through which to view the program and its impact. The results of this 
evaluation provide evidence that, not only does the program have a positive impact 
on teacher and student learning, but also the more faithful the program is to the 
core goals of eMINTS, the stronger that impact is. 
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